NASA Contractor Report 181827 


FORMAL VERIFICATION OF 
AI SOFTWARE 


(NASA-CR-181827) 


P CSCL 09B 


G3/61 


N89-24811 


tJnclas 

0217233 


John Rushby, R. Alan Whitehurst 
SRI International 
Computer Science Laboratory 
333 Ravenswood Avenue 
Menlo Park, CA 94025 

NASA Contract 18226 (Task 5) 
February 1989 


IWNSA 

National Aeronautics and 
Space Administration 

Langley Research Center 

Hampton, Virginia 23665-5225 



Abstract 


The application of formal verification techniques to AI software, particularly 
expert systems, is investigated. Constraint satisfaction and model inversion 
are identified as two formal specification paradigms for different classes of 
expert systems. A formed definition of consistency is developed, and the 
notion of approximate semeuitics is introduced. Examples are given of how 
these ideeus can be applied in both declarative and imperative forms. 
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Chapter 1 

Introduction 


Almost all computer software harbors faults; those faults can lead to failure, 
and failure can have serious consequences. Stringent measures are (or should 
be) taken to ensure that software faults in critical systems will not lead to 
failures that could endanger life or national security, cause harm to the 
environment, or have other undesired consequences disproportionate to the 
benefits provided by normal operation. The goal in software development 
for critical systems should be to develop dependable software — software for 
which justifiable reliance can be placed on the quality of service it delivers. 

There are two main approaches for achieving dependability: fault exclu- 
sion, which aims to prevent the occurrence of faults, and fault tolerance, 
which seeks to detect and recover from the manifestations of faults before 
they can lead to failure. 1 Fault exclusion techniques include systematic 
design methodologies intended to prevent faults from being introduced in 
the first place, combined with thorough testing and examination procedures 
intended to identify and eliminate those faults that slip through the first 
stage. Fault tolerance is based on redundancy and run-time checking; be- 
cause all software faults are design faults, the redundant components cannot 
be simple replicates of the original, but must be “diverse designs” [6]. 

Both fault exclusion and fault tolerance techniques can benefit from the 
observation that dependability is not the same as reliability: it may not be 
necessary to exclude or tolerate all faults, but only those with unacceptable 
consequences [35]. That is to say, dependability can be regarded as reliability 
with respect to unacceptable failures. 

1 What we have called fault exclusion is generally termed fault avoidance; we prefer the 
former term since it has a more active and positive connotation. 
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Dependability requirements for life-critical systems (such as fly-by-wire 
digital control for passenger aircraft) are onerous. For example, the FAA 
requires failures that could endanger a civil aircraft to be “extremely im- 
probable” and “not expected to occur within the total life span of the whole 
fleet of the model” — the FAA has suggested that this should be interpreted 
as a probability of less than 10~ 9 in a 10 hour flight, or 10 -1 ° per hour 
of flight. Substantiating reliabilities such as these through testing is infear 
sible. Furthermore, experimental and theoretical results cast doubt on the 
failure-independence assumption that underlies the belief that software fault 
tolerance techniques can deliver the required dependability [12, 21, 32, 37]. 


1.1 Formal Verification 

Formal verification is a technique that can, in principle , guarantee the ab- 
sence of faults. Formal verification demonstrates consistency between two 
different descriptions of a program. Often, one description is the program 
code itself and the other is its specification, although the method can be 
applied equally well to two specifications at different levels of detail. For- 
mal verification treats specifications and programs as formal, mathematical 
texts, and the demonstration of consistency between them takes the form 
of a mathematical proof. Formal verification steps can be “stacked” on top 
of each other, so that a hierarchy of increasingly detailed specifications car- 
ries the demonstration of consistency all the way from a highly abstract 
“requirements” statement down to executable code. This guarantees the 
“correctness” of the executable code provided: 

1. The notion of consistency that is employed is appropriate for the in- 
tended interpretation. 

2. The requirements statement accurately captures the real (dependabil- 
ity) requirements on the system. 

3. The meaning ascribed to the program code during its verification ac- 
curately reflects its behavior during execution. 

4. The proofs are performed without error. 

Consistency is usually equated with logical implication: it is proved that 
the properties of a lower level imply those of the upper one. In other words, 
it is proven that the lower levels do at least that which is specified in the 
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upper levels. This excludes faults of omission, but not necessarily those of 
commission. 

A requirements statement is just a set of marks on paper; there can 
be no conclusive demonstration that it is an accurate formulation of the 
dependability objectives for the system. The best that can be done is to 
employ notational formalisms that are expressive, precise and perspicuous, 
and to subject the requirements statement to considerable scrutiny. This 
may take the form of testing, either of the requirements specification itself 
(if the specification language is executable), or of a simulation or “rapid 
prototype,” or of analytic scrutiny. The latter can be “formal testing” that 
involves proving conjectures (“if the requirements are right, then I ought to 
be able to prove that the following must be true”), checks for consistency 
and completeness, and informal review. 

The problem of ensuring that the meaning ascribed to a program dur- 
ing its verification (i.e., its formal semantics) accurately reflects its behavior 
during execution (i.e., its operational semantics) is a difficult one. One ap- 
proach is to carry the formal verification down through the specification and 
implementation of the programming language itself, its support software, 
and the hardware on which it runs until the bottom-level assumptions are 
descriptions of elementary logic gates. There has been substantial progress 
recently on each of these levels [8, 14, 15, 30, 39, 40], and some pioneer- 
ing demonstrations that they can be “stacked.” For the near-term future, 
however, it is reasonable to assume that the correctness of progr ammin g 
language implementations will be subject only to informal arguments. 

It is in order to avoid the fourth class of error, errors made during logical 
reasoning, that truly formal, and mechanically assisted, verification is gener- 
ally recommended. The theorems that need to be proven during verification 
are numerous, massively detailed, and of little intrinsic interest. Experience 
has shown that semiformal proofs, as employed in mathematics journals, are 
unreliable when applied to theorems such as these. Consequently, the use 
of mechanical theorem proving (or proof checking) is attractive. 

The inherent limitations to formal verification identified above are likely 
to be exacerbated in practice by compromises (typically incomplete appli- 
cation, or verification of only limited properties) imposed by technical or 
resource limitations. Nonetheless, formal verification should be considered 
an important component in the demonstration of dependability for critical 
systems, and an even more important component in the process of con- 
structing such systems. When integrated with the design and development 
process (rather than being an after-the-fact analysis) , formal verification 
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can contribute significantly to the development of dependable systems by 
encouraging systematic and conscious thought, and a preference for simple 
and perspicuous design and implementation. 


1.2 AI Software and Expert Systems 

This report is concerned with the application of formal verification tech- 
niques to AI software in general and expert systems in particular. By AI 
software we mean software that uses techniques from the field of Artificial 
Intelligence; Genesereth and Nilsson [24] give an excellent modern introduc- 
tion to such techniques. Expert systems can be characterized as AI software 
that use highly problem-specific information (as opposed to general laws) 
in order to achieve performance comparable to human specialists in limited 
problem domains. Expert systems employ “surface” rather than “deep” for- 
mulations of knowledge; they codify empirical associations, special cases, 
and exceptions. These are often expressed in the form of “if-then” produc- 
tion rules, giving rise to the “rule-based expert systems” that are becoming 
widespread. A good survey of expert systems is given by Buchanan and 
Smith [11]; Harmon and King [28] provide a more elementary overview. 

On the face of it, expert systems seem unlikely candidates for formal 
verification. In the first place, their requirements statements are notoriously 
vague: “build a system to do what Bill does” is not a gross parody of the 
specification for an expert system. Even when requirements are specific, 
they may not be amenable to formal analysis. For example, “build a system 
for loan approval that delivers no more than 4% bad loans by value” is not 
a requirement that can be verified analytically. 

Second, the developmental process used for expert systems is largely 
experimental in nature; the knowledge base is refined “in response to errors 
exhibited in solving test cases” [11, page 42]. The hierarchy of specification 
and design that is the foundation of systematic engineering for conventional 
software, and of its formal verification, is absent for most expert systems. 
As Buchanan and Smith observe [11, page 47]: 

Often when one begins designing an expert system, neither the 
problem nor the knowledge required to solve it is precisely speci- 
fied. Initial descriptions of the problem are oversimplified, so the 
complexity becomes known only as early versions of the system 
solve simple versions of the problem. Expert systems are said to 
approach competence incrementally. 
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Parnas puts it less sympathetically [43]: 

The rules that one obtains by studying people turn out to be 
inconsistent, incomplete, and inaccurate. Heuristic programs are 
developed by a trial-and-error process in which a new rule is 
added whenever one finds a case that is not handled by the old 
rules. This approach usually yields a program whose behavior is 
poorly understood and hard to predict. 

Third, conventional verification is concerned with algorithmic software. 
The premise of conventional development methodology and of verification 
is that the goal of the exercise is to produce an efficient implementation of 
a systematic method (an algorithm) for satisfying the requirements. Expert 
systems do not satisfy this premise; solutions are found by search rather 
than algorithmically. The knowledge base is not a conventional program, 
but a collection of heuristic information used to guide the search. 

Finally, the languages in which expert systems and their knowledge bases 
are encoded have not been developed with formal analysis in mind. 


Chapter 2 

Requirements and 
Specifications 


The absence of precise requirements and specification documents for much 
AI software reflects the genuine difficulty in stating a priori the expecta- 
tions and requirements for a system whose capabilities will evolve through 
a development process that is partly experimented in nature. However, if 
formal verification is to be applied to AI software, precise requirements and 
specifications are a necessity. We have proposed a possible solution to this 
dilemma in a previous report [48]. 

The basis of our proposal is first to separate the “inherently AF aspects 
of AI software from the more conventional aspects — aspects that should be 
amenable to conventional software engineering and quality assurance. To 
this end, we distinguish two sets of requirements and specifications for AI 
software: the competency requirements and the service requirements. 

Competency requirements pertain to those dimensions of the overall re- 
quirements that concern “knowledge” or appeal to comparison with human 
skills. We accept that such requirements may of necessity be vague and 
incomplete. Service requirements cover all other requirements and should 
be amenable to statements no less rigorous and formal than those for con- 
ventional software. Service requirements should include descriptions of the 
input and output formats expected, the processing rate, the explanation far 
cilities required, and so on. Service requirements and their decomposition 
through levels of specification should be traceable, verifiable, and testable 
just like those of conventional software. 
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The present investigation is concerned with verification of competency 
requirements. Because we have already conceded that these may be vague 
and incomplete, it may seem contradictory to talk about their formal ver- 
ification. However, it is not necessary to verify full functional correctness 
in order to accomplish something useful. For many critical applications, it 
is not necessary that the system should not fail, only that it should not 
fail “badly.” Accordingly, we further subdivide competency requirements 
into “desired” and “minimum” requirements. The desired competency re- 
quirement will often be defined relative to human expertise and describes 
how well the system is expected to perform. The minimum competency 
requirement, on the other hand, defines how badly it is allowed to perform. 

Whereas desired competency requirements may be hard to state explic- 
itly, there is some hope that minimu m competency requirements may some- 
times be capable of precise definition. For example, one of the expert systems 
developed at SRI, by the Information Sciences and Technology Division, is 
the Automated Air Load Planning System — AALPS [5]. It is used to pro- 
duce schedules and layouts for loading army divisions and their equipment 
(tanks, helicopters, etc.) onto air transports. One of the things to be consid- 
ered is the unloading of the aircraft, especially if this is to be performed in 
flight, using parachute drops. As the heavy equipment is first moved around 
inside the aircraft prior to being dropped, and then actually dropped, the 
aerodynamic stability of the aircraft must remain undisturbed. Specifying 
the desired competency of AALPS is obviously difficult — we want a near- 
optimum loading plan, but mi optimum loading plan is hard to define, and 
it is even harder to determine whether a given plan is optimal. But the 
minimum competency requirement can be given quite a sharp statement: 
the aerodynamic trim of the aircraft must remain within certain limits at 
all times during all stages of unloading in flight. 

We believe that minimum competency requirements may be able to cap- 
ture certain desired “safety” properties for AI software and that such re- 
quirements can often be cast as formal requirements specifications. In the 
following sections we will look more closely at the formal definition of min- 
imum competency requirements for different classes of AI software. 


2.1 Safety and Liveness Properties 

All extensional properties of programs are conjunctions of safety and live- 
ness properties [4]. A liveness property asserts that something good must 
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eventually happen; termination is the archetypal liveness property. A safety 
property asserts that some bad thing does not occur during execution. Par- 
tial correctness is the archetypal safety property: the “bad thing” that must 
not occur is termination with the wrong answer. The term “safety property” 
is a technical one; such properties need have nothing to do with safety as it 
is commonly interpreted — though we believe that many safety properties in 
the conventional sense of the term can be captured formally by properties 
satisfying its technical sense. 

We can formalize these notions as follows. Let I and O be the input and 
output domains of the program P. The input/output behavior of P can then 
be represented by a relation on I x O: P(i, o) will be true if execution of P 
can terminate with output o € O when given input * € I. 1 P may not be 
intended to work for all possible inputs, so we let V be a predicate on I that 
identifies “valid” inputs. If we now let C be a predicate on O that identifies 
outputs satisfying the desired safety property, then the specification that P 
satisfies C for inputs satisfying "V is simply: 2 3 

V* € I,o GO, V(») A P(i, o) D C{o) 


2.2 Classification of Expert System Applications 

Applications of expert systems can be divided into those concerned with 
problems of interpretation (analysis) and those concerned with problems of 
construction (synthesis). Buchanan and Smith [11, pages 28 and 29] tabulate 
some of the major examples of each kind. Among the most common of 
the analytic systems are those concerned with equipment monitoring, fault 
diagnosis, and the screening and interpretation of data. Expert systems 
for synthesis include those concerned with planning, scheduling, loading 
(AALPS is in this category), configuration, and design. We believe that 
another way of looking at this dichotomy between analytic and synthetic 
expert systems sheds more light on their possible requirements specifications. 
Our alternative dichotomy is that between constraint satisfaction and model 
inversion. 


1 We use relations rather than functions (i.e., we do not write P(i) = o) to allow the 

possibility of nondeterministic behavior. 

3 We use -i, A, V, and D to denote negation, conjunction (“and”), disjunction (“or”), 
and implication, respectively. 
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2.2.1 Constraint Satisfaction Problems 

A constraint is a predicate over the output variables of a program. The 
constraint satisfaction problem is to find an assignment of values to those 
variables that satisfies the constraint. The important point about constraint 
satisfaction problems is that we can check whether purported solutions do 
indeed satisfy the constraint. Optimization problems are a variation on con- 
straint satisfaction. We want a solution that not only satisfies the constraint, 
but also maximizes (or minimizes) some function. It may not be feasible to 
check whether a purported solution to an optimization problem is indeed op- 
timal, but it is at least possible to check whether it satisfies the constraint. 
Often, truly optimal solutions are not necessary; in these cases bounds on 
acceptable values may be characterized as part of the constraint to be satis- 
fied (e.g., “find a loading plan that uses no more than three aircraft”). We 
will call these bounded constraint satisfaction problems. We contend that 
minimum competency requirements for synthetic expert systems can often 
be formulated as bounded constraint satisfaction problems — and specifica- 
tions for such problems can be formalized as safety properties. For example, 
let P be the AALPS program, V a definition of “valid AALPS input” and 
C a formalization of the constraint “the loading plan places the center of 
gravity of each aircraft within acceptable limits.” Then the simple safety 
property 

Vi €l,oeO, V(i) A P(i, o) => C(o) 

expresses the requirement that if AALPS is given valid input, and if it 
terminates, then any loading plan produced must place the center of gravity 
of each aircraft within acceptable limits. 

2.2.2 Model Inversion Problems 

A model is a “a simplified representation or description of a system or com- 
plex entity, especially one designed to facilitate calculations or predictions” 
(Collins English Dictionary). 

Computer models can range from systems of equations capable of yield- 
ing highly accurate and precise numerical predictions (e.g., for orbital me- 
chanics) to descriptions that provide only qualitative information. To qualify 
as a model, a system representation or description should have some explana- 
tory or causal value: it should in some sense describe how the system being 
modeled actually works. A model has much in common with a physical 
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theory, and need not be correct or complete in order to be useful — so long 
as its predictions are adequate for their purpose. 

Models have parameters — quantities whose values differ from one in- 
stance of the model to another. In the case of a model for an automobile 
electrical system, for example, the parameters might include the voltage at 
the battery, and whether or not the starter motor is broken. Usually, cer- 
tain parameters can be identified as inputs (i.e., their values are determined 
by factors external to the system being modeled), and others as outputs. 
Inputs cam be regarded as “causes” and outputs as “effects” ; the model is 
then a mechanism for reasoning from causes to effects. 

Computer models are often used in this predictive manner: given values 
for the input parameters, the model is used to predict values for the output 
parameters. Such predictions are usually determined by direct calculation 
and are performed by conventional software. 

The inverse problem — that of finding causes that explain observed effects — 
is what we call the problem of “model inversion.” Whereas problems of 
prediction can usually be solved by direct calculation, problems of model 
inversion must often be solved by search — different values for the input pa- 
rameters are tried in turn until a configuration is found that leads to a 
prediction matching the observed behavior. Of course, such explicit search- 
ing will often be unacceptably slow, so heuristic techniques may be employed 
to perform the search in {in “intelligent” manner. Such heuristic techniques 
can be used in two ways: either they can be used in conjunction with the 
explicit model, or they can be used instead of it. In the first case, heuristics 
can be used to suggest values for the input parameters, the output values 
can be computed, and their divergence from those actually observed can 
be used by the heuristics to adjust the input parameters. (Of course, more 
sophisticated arrangements may sometimes be possible, in which the heuris- 
tics are more intimately connected to the structure of the model.) In the 
second case, heuristics are used to suggest values for the input values and 
these are accepted without further checks. 

We contend that many, if not most, analytic expert systems can be 
characterized as heuristic model inversion procedures of this second kind — 
except that the model that they invert is never constructed explicitly. We 
further contend that the way to produce precise specifications for such expert 
systems is to construct the missing model. 

We can then say that an analytic expert system is consistent with its 
model if the former is an inverse of the latter — that is, if outputs of the 
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expert system, when supplied as inputs to the model, generate predicted 
effects that match those observed. 

This can be expressed formally as follows. Let C and E be the domains 
of “causes” and “effects,” respectively. Let X, a predicate on C x E, be 
an explicit model (from causes to effects), and let P, a predicate on E x C, 
be an expert system (that reasons from effects to causes). Then the expert 
system P is consistent with the model At if 

Vc € C,e€ E : P(e,c) D ,M(c,e) 

It is easy to construct variations on this definition. For example, if □ is a 
relation on effects, with the interpretation that <? □ e means the predicted 
effects e' do not contradict observed effects e, then we can define a weaker 
notion of consistency as follows: 

Vc € C, e € E : P(e, c) D 3 e‘ : M(c, e') A e' □ e 
We can also say that P is complete with respect to M if 
Vc6C,e€S: M(c,e) D P(e,c ) 

Consider an expert system to diagnose faults in the electrical subsystem 
of an automobile, for example. This will typically be a rule-based system 
containing rules such as 

if the starter motor turns 

and there is no spark at plugs 
then 

there is a problem in the HT Coil 

An explicit model, on the other hand, would link the components of the 
subsystem in cause-effect relationships in some suitable degree of detail. A 
highly detailed model might account for actual voltage, current, and resis- 
tance values, whereas a qualitative model might simply deal with the signs 
(i.e., positive or negative) of various quantities and their derivatives. For 
some diagnostic problems, a simple model of fault propagation is all that is 
needed (i.e., a component fault is assumed to affect all sensors “downstream” 
from that component). A simple model for the automobile electrical system 
might note that the battery is “upstream” from both the starter motor and 
the HT coil, and the HT coil is upstream from the spark plugs. (This sort 
of model is sometimes called a “causal net”). The expert system rule given 
above is consistent with this model. The rule 
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if the starter motor does not turn 
then 

there is a problem in the battery 


is weakly consistent with the model if we define e 1 □ e to mean that the 
observations predicted abnormal in e' are a superset of those observed ab- 
normal in e. 

2.2.2.1 Abduction 

There are many different representations for the many different types of 
models — differential equations, directed graphs, and predicate calculus, for 
example. In the case of predicate calculus representations, model-inversion 
can be described by a form of logical inference known as "abduction.” Given 
the premise (Vs : P(x) D Q[x)), the familiar process of deduction allows us 
to reason from the observation P(a) to the conclusion Q(a). Abduction , on 
the other hand, allows us to reason from the observation Q(a) to the possible 
"explanation” P(a). With a complex model, there can be many abductive 
explanations for a given observation and various criteria have been devised 
for selecting a preferred explanation [52]. 

Abduction has been proposed as a technique for model-based diagno- 
sis [17, 41] and methods for mechanizing abductive inference have been 
developed [16, 45, 52]. We believe that abductive techniques can also serve 
to specify minimum competency requirements for certain model inversion 
problems. 

2.3 Formal Requirements Specifications 

We have introduced two classes of problems and shown how formal speci- 
fications can be given for each. We contended that constraint satisfaction 
could provide formal specifications for expert systems that perform synthesis 
while model inversion serves that purpose for analytic systems. Of course, 
this dichotomy is a little simplistic: some aspects of analytic expert systems 
may be best specified in terms of constraint satisfaction, while synthetic 
systems may have some model inversion attributes. And some requirements 
may lend themselves to neither constraint satisfaction nor model inversion 
formulations. (For example, planning probably requires the introduction of 
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a third problem class.) Nonetheless, we believe that our two classes of prob- 
lem formulation are adequate to specify the requirements of many expert 
system applications. 

The question remains, do our formulations make “good” requirements 
statements? One of the principal characteristics desired of a requirements 
formulation is that it should be comprehensible: those responsible for the 
procurement and quality assurance of a system must be able to scrutinize its 
requirement statement and convince themselves that they truly understand 
its implications and limitations. We believe that the statements that any 
results produced by an expert system are to be consistent with “this set of 
constraints” or “that model” do satisfy this desideratum. A model that pro- 
vides some plausible and systematic causal connection between inputs and 
outputs is far more comprehensible and defensible — and surely represents 
more “knowledge” — than a set of associations encoded as rules. 

If constraints and models can serve so well as the requirements specifi- 
cations for expert systems, why do we build rule-based expert systems at 
all? Why not build constraint satisfaction and model inversion procedures? 
We are convinced that the development of such procedures is the correct 
direction to pursue if reliable expert-like systems are to be produced. There 
are, in fact, substantial bodies of work on constraint satisfaction (the book 
by Leler [34] is a good introduction) and qualitative modeling (see the book 
edited by Bobrow [10]). Diagnosis of physical systems is a field in which 
direct model-based procedures have been studied extensively (see Reiter’s 
work [46], for example, and the survey by Davis and Hamscher [18]). 

In the short term, however, it is likely that expert systems will be able to 
outperform systems based on direct constraint satisfaction or model inver- 
sion. For one thing, many problems of interest lack good models (strategic 
warfare and medical diagnosis, for example) and a good ad hoc expert sys- 
tem may be able to do better than a simplistic model or constraint-based 
system. However, we believe that even simplistic models and constraints 
serve to establish useful minimum competency requirements for such expert 
systems. 



Chapter 3 

Languages 


The languages in which expert systems are commonly written, whether 
“rule-based,” “frame-based,” or “object-oriented” have not been developed 
with formal analysis in mind. In particular, they lack formal semantics, and 
generally lack the regularity and conceptual simplicity that would make the 
construction of formal semantics feasible. 

For concreteness, we will use forward-chaining production rule languages 
as our paradigm for languages used in the implementation of expert systems. 
These languages (for example, OPS5 [23]) are appropriate when all input 
data are known in advance, or are easy to collect, and when there are rela- 
tively few hypotheses to be explored. Such are likely to be the case in many 
near-term space and aviation applications of expert systems, where input 
data are usually provided by sensors. 


3.1 Declarative Semantics 

There is some similarity between rule-based notations and logic program- 
ming languages such as Prolog [13] and OBJ [26]. A pure logic programming 
language (e.g., OBJ) has a declarative semantics: a program can be inter- 
preted as a theory in the logic, and computation is equivalent to deduction 
in that logic. The behavior of programs in execution can be predicted by 
proving theorems within the logic. Impure logic programming languages 
(e.g., Prolog) compromise the declarative semantics in order to improve 
performance and to provide for the deliberate “control” of execution behav- 
ior. Thus, most Prolog interpreters omit the “occurs check” (which renders 
them unsound), use depth-first search (which renders them incomplete), add 
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“negation as failure” (whose logical interpretation is a controversial topic 
[22]), and provide extrarlogical “control” constructs such as “cut,” “assert,” 
and “retract.” (Although these features destroy hopes for a declarative se- 
mantics, it is possible to give them a denotational semantics [19].) 

Production-rule languages add further control and other features (e.g., 
nonmonotonicity) that take them even further away from pure logic pro- 
gramming languages — and compromise still further any declarative seman- 
tics. Buchanan and Smith justify this as follows [11, page 34]: 

Many designers of expert systems are uncomfortable with math- 
ematical logic as a representation language because it lacks ex- 
pressive power. Numerous extensions must be made to express 
some of the concepts that are frequently used in applications: 
uncertainty, strategy knowledge, and temporal relations. Some 
logicians are uncomfortable with reasoning that is not theorem 
proving and with knowledge bases that are not axiomatic systems 
that allow proofs of consistency and completeness. The search 
for new logical formalisms that are more powerful than predicate 
calculus reflects the tension between simple, well-understood for- 
malisms and expressive power. 

3.2 Operational Semantics 

If the need to consider control strategies and working memory makes the 
search for declarative semantics for rule-based languages rather unpromising, 
perhaps it will be better to consider more operational semantics — that is, 
semantics that explicitly consider the existence of “state” and control flow. 

Hoare introduced a notation and deductive system for reasoning about 
the partial correctness of imperative programs in his seminal paper [29]. 

In this notation, if C is a program, and P and Q are conditions on 
the program variables used in C, then the notation {P}C{Q.} expresses the 
safety property that if execution of C begins in a state (i.e., an assignment 
of values to program variables) satisfying P, and if execution terminates, 
then it must do so in a state satisfying Q. 

An alternative notation that makes the system state explicit is sometimes 
preferable. In this notation, conditions are modeled as predicates on states 
(assignments of values to program variables), and programs as relations on 
states. If s and t are states, then P (s) is true if state s satisfies condition P , 
and C(s,t) is true if program C, when started in state s, can terminate in 
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state t. The connection between the two notations (see, for example, [27]) 
is given by 

= :/>(•) A C(«,t)3 2(t) 

“Proof rules” for the elementary constructs of a programming language, 
together with rules for their combination, allow properties of imperative 
programs to be proved. An example of a proof rule is that for the “while” 
loop: 

{P aB)C{P) 

{P} while B do C{P A^B} 

The hypotheses of the rule are written above the line, the conclusions are 
written below it. A formula P such that {P A B)C{P) is called an invariant 
of C for B. 

The difficulty in applying these ideas to production-rule languages is that 
the control structures of production-rule systems are less explicit and reg- 
ular than “while” loops, and the control (i.e., conflict resolution) strategies 
used are quasi-nondeterministic. It is possible to give Hoare-style semantics 
for pure nondeterministic constructs (e.g., for Dijkstra’s guarded command 
language [20] 1 ), but the strategies used in production-rule languages are so 
operational that they defy tractable formalization. 

It is important to understand just how complex some of these control 
features are; a brief description of the operation of OPS5 should make it 
clear. 

OPS5 programs consist of rules called productions, each of which com- 
prises a condition part (called the LHS) and an action put (called the RHS). 
Programs operate on a globed database called working memory by repeat- 
edly performing a sequence of actions called the recognize- act cycle: 

Match: eveduate the LHSs of the productions to determine which are sat- 
isfied by the current contents of working memory (the set of eligible 
productions so formed is called the conflict set). 

Conflict Resolution: select one production from the conflict set; if the 
conflict set is empty, return control to the user. 

Act: Perform the actions specified in the RHS of the selected production 
(this is called firing the production). 

‘Dijkstra uses “weakest preconditions” rather than Hoare logic, but the differences 
between these formalisms are not significant to our discussion. 
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The elements of the conflict set are called instantiations; they consist of a 
production and a binding of working memory elements to the variables in 
its LHS. The conflict resolution strategy is the following 2 : 

1. Discard from the conflict set those instantiations that have already 
fired. (The definition of “already fired” is complex; roughly speaking, 
it means that there is some n such that this instantiation has been 
present in every one of the previous n conflict sets, and that it was 
selected for firing in one of those sets.) 

2. Compare the recencies (the cycle on which they last changed) of the 
working memory elements matching the first condition elements in 
each instantiation. Retain those with the most recent elements. 

3. Order the instantiations on the basis of the recencies of the remaining 
working memory elements using the following algorithm to compare 
pairs of instantiations. First compare the most recent elements from 
the two instantiations. If one is more recent thatn the other, the in- 
stantiation with that element dominates. If the two sire equally recent, 
compare the second most recent elements from each instantiation, and 
so on, until either a difference is found or one instantiation runs out of 
elements. If one instantiation exhausts its elements before the other, 
the unexhausted instantiation dominates. 

4. If no single instantiation dominates all others under the previous rule, 
use the one with the most conditions in its LHS; if that does not select 
a unique rule, choose one arbitrarily. 

Forgy [23] defends these rules on the grounds that “they make it easy to 
add productions to an existing set and have the productions fire at the right 
time, and because they make it easy to simulate common control constructs 
such as loops and subroutine calls.” Anyone concerned with formal rea- 
soning (or just reasoning) about rule based systems might wish that control 
structures were explicit rather than “easy to simulate.” (The same argument 
applied to imperative programming languages would justify the elimination 
of all control structures except the “goto,” since it can also simulate those 
control structures.) 

Additional conflict resolution strategies that are commonly employed 
include giving preference to rules according to their position in the rule- 


3 This is the strategy called MEA; OPS5 also has a strategy called LEX — see [23]. 
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base (early rules have preference), or associating explicit priorities with rules 
(rules with high priority have preference). 


3.3 Approximate Semantics 

The inescapable conclusion from the previous two sections is that the con- 
struction of either declarative or Hoare-style semantics for current rule-based 
languages is a hopeless task. In the long term, we expect that concern for 
predictability and reliability will lead to the development of programming 
languages with tractable semantics for expert system applications (just as 
it has, to at least some extent, with conventional languages) . In the near 
term, however, we either have to accept something less than perfect, or give 
up. 

Because we have already abandoned the attempt to prove correctness 
for expert systems, being willing to settle instead for proofs of, possibly 
modest, safety properties, it may be that merely “approximate semantics” 
will suffice for our purposes. 

Recall that the (exact) functional behavior of a program P is identi- 
fied with the relation P(i, o). An approximate semantics will associate a 
somewhat different relation Papprox(*\ o). To be useful, the approximate se- 
mantics must be conservative — that is, they must not “underestimate” the 
exact behavior (at least when applied to valid inputs). Thus we require: 

V(0 A ^approx (*) °) 3 P{*, o) 

It will then follow that safety properties established with respect to the 
approximate semantics will hold for the exact semantics as well. 


3.3.1 Approximate Declarative Semantics 

We have seen that rule based languages differ from pure logic programming 
languages most violently in providing implicit and explicit control over the 
order in which rules are selected. (The other serious difference is that rule- 
based languages allow values to be asserted and retracted; we will ignore 
this issue for now.) If there were no conflict resolution strategy — i.e., if any 
and all matching rules could fire — then we would be closer to a purely logical 
interpretation of the rules. Now the effect of conflict resolution strategies is 
to limit which rules can fire — so that fewer things will happen in the pres- 
ence of conflict resolution than in its absence. This means that a logical 
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interpretation of a rule-base (i.e., no conflict resolution) should indeed pro- 
vide a conservative approximate semantics. Unfortunately, it will often be 
so conservative that nothing useful can be deduced. An example might help 
make this clear. 

Consider the following rules from a system to handle hospital admissions, 
and suppose that conflict resolution is performed by giving priority to rules 
according to the order in which they appear. 

if head-injury(X) 

and time -since- in jury(X) < 8 
then 

send-to(X) :* emergency-room 


if can-walk(X) 
then 

send- to (X) :* waiting-room 


Now suppose we would like to establish the safety property “anyone who 
has had a head injury within the last 4 hours will be sent to the emergency 
room.” 

Interpreting the rules as a logical theory, we have 

head-injury (X) A time-since-injury(X) <8D send-to(X) = emergency- 
room 

can-walk(X) D send-to(X) = waiting-room 

It is clear that we can establish our safety property from the first of 
these formulas. However, because there is no ordering implied between the 
formulas when interpreted in logic, “firing” the first does not preclude the 
second one from firing also. Thus, if it happened that patient X was able to 
walk, we might also conclude that he should be sent to the waiting room. 
In approximating our rule base by a logical theory, we have lost too much 
information. For the conflict resolution strategy considered in this example, 
it is easy to see that the strategy can be encoded in the approximating 
theory: we simply conjoin to the antecedent of each formula the negations 
of the LHSs of earlier rules: 
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head-injury (X) A time-since-injury(X) <8 D send-to(X) = emergency- 
room 

->(head-injury(X) A time-since-injury(X) < 8) A can- walk (X) D 
send-to(X) = waiting-room 

The dynamic nature of most other conflict resolution strategies precludes 
such simple static representations, however. 

Although simply regarding a rule base as a logical theory may provide 
an excessively conservative semantics, it may still be possible to derive some 
benefit from the exercise. Suppose we were to modify the theory by adding 
clauses to the antecedents of its formulas until we were able to prove the 
desired safety properties. This would make explicit the requirements on 
the conflict resolution strategy to be employed — which could be valuable 
information. Informal arguments could then be used to justify the differences 
between the rules and the derived formulas (or the rules could be modified 
to make them less dependent on the conflict resolution strategy). 


Consistency 

There is a flaw in this reasoning, however. If the derived theory is inconsis- 
tent, then it can be used to prove any safety property whatsoever. Before 
we proceed to analyze this issue, it will be helpful to review some of the 
notation and terminology of logic. 

A (propositional) theory T is simply a set of (propositional) formulas; a 
formula 7 is a theorem of T if T b 7 — that is, if any model of T is also a 
model of 7. By the deduction theorem, this is equivalent to I- T D 7 — that 
is, T D 7 must be valid, (true in all interpretations). If T is inconsistent, 
then it has no model (by definition), so all interpretations will valuate T to 
false, and hence T D 7 to true — whatever formula 7 may be. 

It seems, therefore, that we must check any theories derived from rule 
bases for consistency before proceeding to draw any conclusions. In fact, 
the situation is rather more complex than this. For the theory derived from 
a rule base to be consistent simply means that it has a model (i.e., some 
interpretation of its constant, function, and predicate symbols that valuate 
all its formulas to true). However, it may be that the facts initially asserted 
into the working memory of the expert system are inconsistent with any 
models of its rule base. Again, we will be able to prove arbitrary formulas. 
The explanation is, of course, that it is not the rule base alone that induces 
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the theory corresponding to an expert system — it is the rule base plus the 
initial facts. 

This explains something that others have noted: the notions of inconsis- 
tency employed in logic and in rule based expert systems do not coincide. 
In logic, the following set of formulas is consistent 

p D q 

p A r D -i q 

because both formulas evaluate to true in any interpretation that assigns 
false to p. However, if these formulas are interpreted as rules, and p and 
r are input variables assigned the value true, then both q and ->q will be 
asserted — an obvious contradiction. The problem here is that although the 
rules are consistent on their own, they form an inconsistent theory when 
combined with certain initial values. That is, the following “instantiated 
theory” is inconsistent: 


P ^ q 

p A r D -i q 

P 

r 

This observation allows us to define consistency for the theory corre- 
sponding to a rule-based system as follows: let Jo be any interpretation for 
the inputs to the system (i.e., assignment of initial values), 3 then there must 
exist a model for the rule base (interpreted as a theory) that is an extension 
to Jo. 

Several authors have proposed methods for testing rule-based systems 
for consistency [9, 25, 42, 53] but none of them present a rigorous definition 
for rule-base consistency, though Ginsberg [25] does observe that it is dif- 
ferent from the notion of consistency in a logical theory (the example above 
is due to him). We therefore believe that ours is the first precise defini- 
tion of consistency for rule-based systems. It should be possible to use our 
definition to provide a proof of correctness for Ginsberg’s KB-Reduction al- 


3 If there is a validity constraint on input values, then we can restrict the interpretations 
to those that satisfy it. 
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gorithm [25] . 4 It should also be possible to accommodate the simpler forms 
of nonmonotonic reasoning (such as the closed-world assumption). 

The notion of consistency derived above is a logical one: it simply as- 
sures us that there is some way of satisfying the rule base given any (valid) 
assignment of values to the inputs, without assuming a particular conflict 
resolution strategy. The “answer” produced by the system will be (part of) 
the interpretation that constitutes a model for the rule base. Now there 
may be several models in general, and so several different sets of outputs 
corresponding to the same inputs. This naturally prompts the introduction 
of a second notion for consistency: if the same set of inputs can produce 
several sets of outputs, then it may be desirable in many cases that each of 
these sets of outputs should be “similar” or “consistent” with each other. 5 
That is, we should require: 

V(») A P(i, oi) A P(i, o 2 ) D 0 !~ 02 

where a denotes “similarity” (presumably an equivalence relation). 

It is not clear how (or whether) one can verify this property of “output 
consistency” in general; it is similar to “sensitivity” or “continuity” — the re- 
quirement that inputs that are close together should generate outputs that 
are also close together. Empirical tests seem the only plausible way to vali- 
date such properties. 6 A property that is sometimes capable of verification is 
the stronger one that the outputs corresponding to any set of inputs should 
be unique. In the case of a rule-based system, this will be so if the order 
in which rules are fired does not affect the final values produced. This lat- 
ter property is generally called the Church-Rosser property. Very effective 
tests for the Church-Rosser property are known for certain restricted log- 
ics (e.g., the Knuth-Bendix procedure [33] for equational theories), together 
with weaker results for more general cases (e.g., [47]). It is possible that 
these techniques also could be adapted to restricted rule bases. 

4 The methods of Suwa et al. [53] and Nguyen et al. [42] are so weak and ad hoc that 
they do not require further investigation. Bezem’s method [9] is simply an algorithm for 
testing whether a particular initial interpretation Io can be extended to a model for the 
rule base. 

5 This notion of consistency is likely to be particularly appropriate for analytic expert 
systems: how much trust should one place in a system that is capable of producing totally 
different answers from the same inputs? 

®If the equivalence classes of the similarity relation can be identified with some pred- 
icate, then the problem becomes one of constraint satisfaction — which we believe is 
verifiable. 
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Another property likely to be of some interest is completeness: a rule- 
based system is incomplete if it can fail to assign a value to some output. 
Analogous to our logical definition of inconsistency, we could say that the 
theory corresponding to a rule base is incomplete if it has models in which 
the interpretation of some output is unconstrained. It seems plausible that 
the KB-Reducer algorithm of Ginsberg [25] can be extended to this case. 

We began with the idea that it might be possible to interpret a rule base 
as a logical theory and thereby deduce some approximate, but conservative, 
properties. We have seen that it is necessary to ensure a strong form of 
consistency for the theory so produced. Given such a consistent theory, it 
may then be possible to prove that certain constraints will be satisfied in 
any execution of the system. Performing such a proof might seem to require 
general theorem proving capability. In fact, all the information required 
is gathered as part of Ginsberg’s KB-Reducer consistency-checking proce- 
dure. KB-Reduction is essentially a form of symbolic execution [31]; for 
each “hypothesis” , “labels” are built up that indicate the combinations of 
input values that will cause that hypothesis to be asserted. The label for 
a hypothesis asserted on the right-hand side of a rule is computed by sym- 
bolic evaluation of its left-hand side. For example, if the “partial labels” for 
hypotheses D and A are respectively p V q and p, then the rule D A ->A — > B 
will add (p V q) A ~>p (which simplifies to q A ->p) to the label for B. Thus, to 
verify a constraint such as “anyone who has had a head injury within the 
last 4 hours will be sent to the emergency room,” we simply check that the 
label for “being sent to the emergency room” is implied by the condition 
“head injury within the last 4 hours.” 

It should be perfectly straightforward to develop an implementation of 
Ginsberg’s KB-Reducer procedure that supports this limited form of verifi- 
cation. It should be noted that there is great similarity between these topics 
and algorithms for rule-based systems and those for decision tables [36, 38] . 

3.3.2 Approximate Imperative Semantics 

While the investigation of approximate declarative semantics has shed some 
light on the question of consistency, and suggests a way of verifying con- 
straints for elementary rule bases, we believe that approximate imperative 
semantics provides a more suitable base for verifying more complex proper- 
ties. As with the declarative approach, the more complex forms of conflict 
resolution will not be modeled, but the presence of an explicit state allows 
operational constructs to be modeled more accurately. In particular, the as- 
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signment of values to variables (and subsequent reassignment of new values) 
is easily accommodated. 

The semantics of an individual rule 

if C then A 

can be modeled by the standard axiom 

{P AC} A{P} 

{P} if C then A{P) 

Here the predicate P is an invariant maintained by the rule. If a num- 
ber of rules all maintain the same invariant, then their nondeterministic 
combination will also maintain that invariant, as will the iteration of that 
combination. 

The basic idea for rule base verification using approximate imperative 
semantics is to establish an invariant for the entire rule base and to verify, 
using the rule above, that it is indeed an invariant for each rule in the rule 
base. If the invariant is also guaranteed true for any valid input, then one 
may conclude that it will be true when the rules terminate. 

It might seem that any formula that is true of the inputs to the system, 
and that is maintained true by all the rules in the system, can only encode a 
very weak property. In a strict sense, this is obviously true, but a great deal 
of information can be encoded in the invariant using control variables. These 
are variables that record and control the progress of computation in the rule 
base. When a rule that initiates a new stage of processing fires, it sets a 
control variable to a particular value to indicate that fact. Some rules will 
condition their LHSs on the values of control variables (e.g., if control-var = 
x and other conditions tben actions), so that they are eligible for execution 
only during certain stages of processing. The system invariant can exploit 
this attribute by having different components for different control values — 
that is, it may have the form 

control-var = x invariant-component-x 

A control-var = y D invariant-component-y 

A control-var = z D invariant-component-z 

Informal arguments will probably have to be used to argue that, on termina- 
tion, the control variable will have a value associated with an “interesting” 
component of the invariant. 
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In the case of constraint satisfaction problems, the constraint will provide 
the invariant, or at least one component of it, to be verified. For model 
inversion problems, the invariant to be verified will be that the components 
of the “answer” being constructed are consistent with the model employed. 
In particular, the rule “if symptoms then cause” should be supported by the 
argument that in the explicit model, “cause” leads to “symptoms.” The next 
chapter presents a preliminary experimental investigation into the utility of 
these ideas. 



Chapter 4 

Examples 


4.1 Introduction 

In this chapter we report on some of our experiences and observations in 
attempting to apply techniques proposed in this report to the verification of 
rule-based expert systems. The experiments described in this chapter should 
not be considered as indications of our current capabilities; rather, they serve 
to illustrate what we believe to be some of the promising approaches and 
interesting challenges concerned with expert system verification. 

These experiments were conducted using various notations and formalisms, 
including the P-BEST expert system shell and the EHDM verification en- 
vironment [55]. P-BEST is a forward-chaining production system that re- 
sembles OPS5 [23] and other similar tools. For a more complete description 
of the P-BEST environment, see Appendix A. EHDM is a formal specifica- 
tion and verification environment constructed at SRI. For an introduction 
to EHDM, see Appendix B. 


4.2 Example— Constraint Satisfaction 


In this section, we consider the application of formal reasoning to the devel- 
opment and verification of constraint-satisfaction type expert systems. The 
example presented here considers the problem of constructing an expert 
system that plays the game “Tic-Tac-Toe.” 
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4.2.1 Problem Description 

The game of tic-tac-toe may be thought of as a constraint satisfaction prob- 
lem. Given an arbitrary legal board configuration, the computer is required 
to place a marker in a position such that a certain goal is achieved. We 
considered the “classical” two-player version of tic-tac-toe as played on a 
3x3 board (illustrated in Figure 4.1). The goal of the game is to place three 


1,1 

1,2 

1,3 

2,1 

2,2 

2,3 

3,1 

3,2 

3,3 


Figure 4.1: Tic-tac-toe Board Configuration 

markers on the board so that they are in alignment either vertically, hor- 
izontally, or diagonally, while preventing the opponent from placing three 
markers in alignment. Play begins with an empty board, and proceeds as 
alternating players place markers on the board until either: (a) some player 
achieves the placement of three markers in a row, or (b) no remaining empty 
positions exist. 

4.2.2 Rule-Based Implementation 

We implemented an initial rule base to play tic-tac-toe. To further constrain 
the problem, we adopted the convention that the opponent would always get 
the first move. In an effort to model the process of knowledge engineering, 
in which insight into the problem space is gathered in a piece-wise fashion, 
we adopted a naive playing strategy that can be informally stated as follows: 

• If the computer can win (i.e., place a marker so that there are three 
markers in a row, either vertically, horizontally, or diagonally), then 
have the computer place the marker in the winning position and ter- 
minate the game. 

• If the computer must block to prevent the opponent from winning, 
then have the computer place a marker in the blocking position. 

• Otherwise, have the computer plaice a marker in one of the following 
positions, arranged according to desirability: center, corner, middle. 



4.2. Example-Constraint Satisfaction 


29 


Using P-BEST, a forward-chaining expert system shell, we defined the 
following fact templates. In P-BEST, a fact template must be declared for 
each possible assertion. In the case of the tic-tac-toe example, there were 
three possible types of assertions that could be made into the knowledge 
base: 


• position assertions 

• turn assertions 

• opponent move assertions 

Position assertions were used to represent the state of the playing board 
at any given point during the game. There were exactly nine such assertions 
at all times — one assertion for each of the nine possible positions on the 
playing board. At the beginning of play, each of these assertions contained 
the designation that the corresponding position was free of any marker. As 
play progressed, the position assertions were altered to reflect when a marker 
was placed in the corresponding location. Such markers were designated as 
“opp” short for opponent, and “com” for computer. 

Turn assertions were used to record whose turn it was at any given point 
in the game and to keep track of how many turns had transpired. 

Opponent-move assertions were used to communicate the intention of the 
human opponent to the knowledge base. Incorporated into the knowledge 
base were several rules that were responsible for querying for the opponent’s 
move, checking its legality, and asserting a proper opponent-move assertion 
into the knowledge base. Because these rules do not effect the behavior of the 
program with respect to its strategic performance, they are not considered 
in the subsequent analysis. 

Figure 4.2, gives an example of what the declaration of these fact tem- 
plates, called “ptypes” or predicate-types in the terminology of P-BEST, 
looks like. For a more complete description of the P-BEST system, see 
Appendix A. 

For an example of the syntax of the P-BEST rules, refer to Figure 4.3, 
which gives an example of the rule “make_random_move” from the tic-tac-toe 
rule base. This rule is given an explicit ranking, in this case -10, which affects 
the selection of which rule to execute when there are multiple candidates. 
The higher the rank of the rule, the greater priority given that rule during 
conflict resolution. The effect of this rule is that when it is the computer’s 
turn to move, if no rules with higher ranks are eligible to fire and if there is 
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(define-ptype POSITION 


“A unique position in the 3x3 board. 1 


row 

col 

marker 


j integer - range 1 to 3. 
; integer - range 1 to 3. 
; free, com or opp. 


Figure 4.2: P-type Fact Template Declaration in P-BEST 

a free position left on the board, the computer will select one of those free 
positions and replace the “free” designation with its marker, “com” . 

The completed initial rule base consisted of 11 rules: 

• 3 rules to detect a condition where the computer can win 

• 3 rules to detect a position where the computer must block 

• 2 rules to recognize lost and tie games 

• 2 rules to get and check human input 

• 1 rule to make a random move as a default case 

plus a handful of LISP functions that assisted in managing the display of the 
game board. The complete code for the rule base is given in Appendix C. 

4.2.3 Requirements 

Although it is sometimes quite difficult to identify expectation for a system 
whose capabilities will evolve through a developmental process, in order to 
validate the system, either through testing or through formal analysis, it is 
imperative that the requirements against which the system is to be evaluated 
be made explicit. 

Tic-tac-toe can be viewed as an optimization problem; given a board 
position, the computer is expected to pick the best move possible. Optimally, 
we would like to have a system which wins; however, in this particular game 
it is a well-known fact that a win cannot be guaranteed. A win occurs 
only when the opponent makes a mistake; if no mistake is made by either 
player, the game terminates in a draw. Therefore, a desired competency 
requirement , which is also a liveness property for this system, might be: 
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(defrule MAKE-RANDOM-MOVE 

“Select a pseudo-random position to place marker. 
Prioritise positions as: 1. center, 2. corners 
and 3. middles.* 
at rank -10 states 
if there exists a turn called tl 
such that player is com and 
count is ?x and 

there exists a position called pi 
such that marker is free 
then forget pi and 
forget tl and 
remember a position 
which inherits from pi 

except that marker is com and 
remember a turn 

such that player is opp and 
count equals (1+ ?x)) 


Figure 4.3: Rule make-random-move From the Tic-Tac-Toe Rule Set. 

• The system wins, when possible. 

This implicitly includes the notion of playing to the standards of a human 
player; it implies that a strategy is pursued that encourages the opponent 
to make an error, and recognizes such situations in order to capitalize on 
mistakes and force a win. Such a notion would be very difficult to capture 
formally or to check with formed analysis. Even in the game of tic-tac-toe, 
which has limited number of possible system states, an optimal move in 
an arbitrary configuration would be difficult to recognize; in some sense it 
would involve recognizing the strategy of the opponent which may not be 
deducible from the board configuration alone. 

A minimum competency requirement, which is also a safety requirement 
for this system, is somewhat more tractable; above all: 

• The system shouldn’t lose. 

We have already noted that it is impossible for an opponent to force a win; 
therefore, the minimum competency we required of our system was that it 
never make a mistake that would result in the opponent’s winning. 
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To formalize this minimum competency requirement, we introduced the 
notion of a safe state. We defined a safe state, which is expressed in terms 
of board configurations, to be those configurations in which the computer 
has won the game, or in which the opponent cannot win the game with the 
placement of the opponent’s next marker. In the language of EHDM, this is 
expressed as: 

safe-def: Axiom 
safe(kbx, s n ) 

•O won_game(kbx, s n ) 

V -do8e_next_move(kbx, s n ) 


Under our initial formulation of rules, there was no reason to believe the 
expert system should be able to win a game of tic-tac-toe because it does 
not attempt to pursue a winning strategy (the computer simply selects a 
pseudo-random move when it cannot immediately win and need not block). 
However, it was reasonable, to hope that the computer would not lose un- 
der this strategy. Specifically, three rules were included in the rule base, 
blk-row, blkjcol, and blk-dia, that prevent the opponent from placing a 
marker in any row, column or diagonal which would result in a lost game. 

4.2.4 Formal Specification 

We constructed a formal specification of the tic-tac-toe rule base and pro- 
posed to verify that specification against the minimum competency require- 
ment that the system must not lose the game. The specifications for the 
tic-tac-toe rule base are contained in Appendix D. In order to create a 
formal specification for this application, it was necessary first to define an 
approximate semantics for the language of our expert system shell, P-BEST. 

We expressed the nonmonotonicity of the rules by introducing an explicit 
notion of system state. As explained in Section 3.3, one way to accomplish 
this in the EHDM language is to add to value-returning functions an extra 
parameter that corresponds to a specific system state. For example, one of 
the modules in the specification describes a formal object called a kb, which 
is declared to be a set of facts (see module kbs in Appendix D). A function, 
kb Just, of two parameters is provided to access the value of objects of this 
type; one parameter is the knowledge base for which to return the value; 
the other is the current system state. Therefore, the declaration for this 
function in the language of EHDM is: 
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kbJnst: functions [kb, state — ► set [fact]] 

With this formulation, we can express that facts that are associated with a 
given knowledge base in a given state may possibly not be associated with 
that knowledge base (i.e., may have been retracted) in some later state, 
without the danger of such retractions leading to logical contradictions. 

Each fact template, or ptype, declaration in the rule base becomes a type 
declaration in the formal specification. For instance, the type position is 
declared to be a subtype of type fact. This allows the declaration and 
use of logical variables that range over all possible positions. Along with 
the type declaration, a function is defined that tests for membership of a 
certain class of instances of a fact in a knowledge base at a particular state. 
With these declarations, a relatively straightforward translation of the rules 
into a logical notation can occur. This can be seen by comparing any of 
the rules in Appendix C with their corresponding formal specifications in 
Appendix D. 

The proof strategy for establishing that the system conforms to the spec- 
ification of our safety property involves simple induction over all valid se- 
quences of moves. In such a scheme, the initial state of the system, in which 
no markers have been placed on the board, is shown to conform to the prop- 
erty in question. In this case, the proof of the safety of the initial state 
is trivial; since there are no markers on the board, there is no way for the 
opponent to win on the next move. Then, given an arbitrary "reachable” 
state that preserves the safety property, if one can establish that all reach- 
able next states also preserve the property, by appealing to induction one 
has established that the property is maintained over all valid sequences of 
moves. 


O 


O 


X 


Figure 4.4: Example Configuration 

In attempting to construct the proof, it soon became apparent that it 
was impossible to prove the safety property without considering the implicit 
control flow in the rule base and the sensitivity of the operational semantics 
to the ordering of assertions. In the tic-tac-toe rule base, for example, the 
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rules dealing with blocking the opponent have precedence over the rule for 
placing the computer’s marker at random. In a situation such as that shown 
in Figure 4.4, where the opponent’s markers are represented as “O” and the 
computer’s markers are represented by “X”, it cannot be proved that the 
computer will not lose on the next move (one of the criteria for the safety 
property in question) if the computer places its marker in any position other 
than row 2, column 1. This is a direct reflection of the fact that it is unsafe to 
execute any rule in the above situation except the rule that explicitly deals 
with blocking an opponent’s win on any particular column; however, if the 
rules are treated as being nondeterministically selected for execution, this 
crucial piece of knowledge is missing and it is impossible to reason formally 
about the safety of the system. 

The informal operational semantics of the tic-tac-toe rule base include a 
ranking of rules. This ranking will prevent the make _random_move rule from 
firing when one of the blocking rules is also a candidate. Since exactly one 
rule fires per state transition, it is operationally impossible for the wrong 
rule to fire. We needed to find a formalism that could capture this implicit 
proceduralism of the rule base. 

Making use of the concept of guarded commands as put forward by 
Dijkstra [20], we adopted a formulation that approximates the proceduralism 
of this application. To do this, we defined a set of operations, one for 
each of the rules, such that the precondition of each operation consisted 
of the conjunction of the negation of the preconditions of all operations of 
higher precedence with the precondition of the corresponding rule. This is 
illustrated in the module guarded.ops in Appendix D. This is only an 
approximation of the actual operational proceduralism because it results in 
a strict ordering of the rules; the actual rule base is only ordered with respect 
to groups of rules at differing precedence levels — rules within each group are 
nondeterministically selected. However, this strict ordering is sufficient to 
establish the safety property in question. 

The operational semantics of the rule base is also sensitive to the or- 
der in which facts are asserted into the knowledge base. For example, the 
make_random_move rule (Figure 4.3) selects a free space to place a marker. 
Contrary to what the name of the rule suggests, the choice of free space is 
not random, but totally deterministic. As the documentation of the rule 
suggests, the center is always chosen if it is free, followed by the corners (be- 
ginning with the upper-right-hand corner and proceeding clockwise), and 
finally by the middles. This precedence occurs because the conflict resolu- 
tion strategy of P-BEST gives priority to rule instances which reference the 
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most recently-asserted facts. In order to capture this operational dependence 
upon the ordering of the initial assertions, additional axioms were introduced 
to make this ordering available during formal reasoning (see module ttt-rlsl 
in Appendix D). 

The proof attempt uncovered four configurations of the board which 
could result in the computer being forced into a no-win situation; these con- 
figurations occur when the opponent can win in two possible ways, making 
it impossible for the computer to block the opponent in a single move. One 
such configuration is illustrated in Figure 4.5. We were able to prove that the 


O 


O 



Figure 4.5: No-Win Sample Configuration 

strategy of the tic-tac-toe rule base prevented the opponent from reaching 
two of these states. However, the formal analysis showed that the remaining 
two configurations could be reached and could cause a loss. Because of the 
existence of these situations, it was impossible to prove the safety invariant 
for the blocking rules as they were originally formulated. In other words, 
our original rule base was incorrect; it failed to consider situations which 
were both reachable and contradictory to our safety requirement. 

With this insight, we were able to expand the definitions of our safety 
requirement to be: 

safe-def: Axiom 
safe(kbx, s n ) 

o- won_game(kbx, s n ) 

V ->(lose_next_move(kbx, e n ) 

V unsafe jconfig_l(kbx, s n ) 

V unsafe _config_2(kbx, s n ) 

Vunsafe_config_3(kbx, s n )Vunsafe_config_4(kbx, s n )) 


To correct the rule base, two additional blocking rules were added at a higher 
priority with respect to the other blocking rules; these rules specifically 
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countered the two remaining unsafe situations. With the addition of these 
two rules, the inclusion of their logical equivalents in the formal specification, 
and the expanded safety definition, it was possible for the proofs to be carried 
forward. Samples of proofs in the methodology of EHDM are included at 
the back of Appendix D. We did not attempt to complete all proofs; it was 
not necessary to formally complete all the proofs, nor to prove the induction 
hypothesis, in order to derive the benefits of the formal analysis. 


4.2.5 Observations 

From this experiment, we are able to draw the following conclusions. 

The translation of expert system rules into logic requires a formulation 
that captures the state-dependent nature of the inference engine. An approx- 
imate semantics for rule-based systems can be formulated which provides an 
adequate framework for reasoning formally about some of the behavior of 
the system. 

While it may be possible to establish very weak invariants independent 
of the conflict resolution strategies involved in the expert system, and inde- 
pendent of the inherent proceduralism involved, it would be impossible to 
establish many important safety properties, such as the one defined for this 
example system, without capturing the implicit control flow. 

Reasoning formally about this application uncovered critical situations 
that were originally overlooked. While the rule base was not incomplete 
or inconsistent according to syntactic analysis, the original rule base was 
incorrect, because it was conceptually incomplete for a class of situations 
that could have led to violation of the safety property during operation. 
Such conceptual incompleteness would not have been discovered through 
any form of syntactic analysis of the rule base. 


4.3 Example— Model Inversion 

In this section we consider the application of formal reasoning to the develop- 
ment and verification of model-inversion type expert systems. The example 
presented here considers the problem of constructing an expert system to 
diagnose problems in a car’s electrical system. 



4.3. Example-Model Inversion 


37 


4.3.1 Problem Description 

As in all diagnosis systems, the goal of our example program is to be able 
to identify the most-likely cause of a problem given a set of observed effects. 
To constrain the problem, we consider only a simplified subset of an actual 
electrical system. Our subset consists of the following components: 

• battery cells 

• battery 

• ignition switch 

• points 

• coil 

• distributor 

• plugs 

• solenoid 

• starter 

We limit the set of potential causes to be the set of failed 
In other words, we would consider the failure of the ignition 
a potential cause for the observation that power is failing to 
to the points and starter solenoid, but we would not consider the state of 
the wires connecting these components as potential sources of the problem. 
Further, we consider only single points of failure in this exercise — for any 
given set of observations, the expert system is required to find the single 
most-probable cause which accounts for the observations. 

4.3.2 The Formal Model 

We created a formal model of our simplified automotive electrical system 
in the language of EHDM; this model is contained in modules components 
and engine-model in Appendix H. The focal point of this model is the ax- 
iom electrical-system_model shown in Figure 4.6. The accuracy of this 
formalization is based on a number of assumptions (e.g., the fact that the 
car in question is not running); however, the model is adequate for the illus- 
trative purposes of this discussion. The model asserts that there is a causal 
relationship between various components of the system. To have power at 
the ignition switch, for example, the model states that it is necessary that 
the battery be in a good state of repair and that the battery be delivering 


components, 
switch to be 
be delivered 
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electrical jystem_model: Axiom 
(power.at (battery) O good (cells)) 

A (power.at (ignition) -O- power.at (battery ) A good (battery)) 

A (power.at(points) power.at (ignition) A good (ignition)) 

A (power.at(coil) O power. at (points) A good(points)) 

A (power.at (distributor) power.at (coil) A good(coil)) 

A (power.at(plugs) O power.at (distributor) A good (distributor)) 
A (power.at(solenoid) power. at (ignition) A good(ignition)) 

A (power.at(starter) O power.at (solenoid) A good (solenoid)) 


Figure 4.6: Model of Simplified Automotive Electrical System 


power. This is representative of the fact that the battery is “up-stream” 
from the ignition in our simple model, just as the ignition is up-stream from 
both the points and the starter solenoid. 

4.3.3 Rule-Based Implementation 

For this experiment, we developed a small backward-chaining rule inter- 
preter, which is a variation on Winston’s expert problem solver [56]. Backward- 
chaining systems start with an unconfirmed hypothesis and attempt to con- 
firm it. This involves identifying rules that substantiate the hypothesis, and 
then attempting to verify the conditions upon which the applicable rules are 
dependent. Apart from some syntactic sugar in the form of extensions to 
simplify the expression of rules and hypothesis, there are two major differ- 
ences between Winston’s original system and our enhanced version: 

1. Our system allows the appearance of “not” clauses in the LHS of rules 

2. Our system allows the specification of questions to be used to obtain 
observations from the user, rather than attempting to synthesize such 
questions from the symbolic representation of the facts 

We created two separate implementations of this rule base. One imple- 
mentation was distilled from a set of troubleshooting and diagnosis proce- 
dures contained in Chilton’s Import Car Repair Manual [54]. We would 
expect the quality of this rule base to approximate the results of most stan- 
dard knowledge engineering efforts. A subset of the rule base is given in Ap- 
pendix F. The other implementation was derived from the abstract model 
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of our simplified system through a relatively mechanical process. This rule 
base is given in Appendix G. 

The procedure for generating the rule base from the axiomatic specifi- 
cation of the model, as seen in Figure 4.6, was straightforward. For each 
equivalence relation specified by the model, a rule is generated to capture 
the equivalence. For example, the relation 

(power_at (starter) power.at (solenoid) A good(solenoid)) 


caused the following rule to be included in the rule base: 


(define -rule d7 

(if (power at solenoid) 

(good solenoid)) 

(then (power at starter))) 

Then, rules were provided which define the possible observations and pro- 
vide questions that may be asked of the user to acquire information about 
the observations. Finally, a set of rules were added which identified the 
global dependencies in the system; these rules serve to associate the failure 
hypothesis with the observations about system behavior. 

4.3.4 Requirements and Validation 

As suggested in Section 2.2.2, requirements for expert systems which have 
formal models can be expressed as 

Vc e C,e € E : P(e,c) D M(e,e ) 

where C is the set of all causes, and E is the set of effects. This leads to 
a procedure for validating the operation of the expert system against the 
expectations of the model for any given set of causes and effects. 

Our simplified model identifies eight possible causes for the system to 
fail; these eight causes correspond to the failure of one of the eight system 
components. The effects in our model are limited to an observation of where 
power is available and where it is not. There is an assumption that the failure 
of a particular component causes the flow of power through that component 
to cease, making the power unavailable to all components down-stream from 
the failing component, and that the presence or absence of such power is 
observable. What we were interested in showing was, that given a certain 
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set of observations presented to the expert system, an answer would be 
produced which, when presented to the model, would cause the correct set 
of effects to be predicted. 

To allow the model to be used to make such predictions, we defined two 
additional EHDM modules, test and predict (see Appendix H), which we 
used to predict the values of the observable effects. To establish which effects 
should be observable according to the model, we set the constant comp in 
module predict to the value of the failed component. We then attempted to 
prove eight lemmas, each of which assert that power will be visible flowing 
through a particular system component. The proofs of these lemmas are 
derivable from the characteristics of the model, the value of comp, and the 
assumption that there is a single point of failure. Those lemmas for which 
the proof succeeds correspond to the set of predicted effects. 

Consider the example of a failed ignition switch. To select the set of 
components through which the model predicts power will flow, we set 

comp = ignition 


and invoke the EHDM provemodule command. The results of this command 
is shown in Figure 4.7. In this case, the model predicted that there would 


Proof summary for module predict 


power_at_star UNPROVED 
power_at_sole UNPROVED 
power.at.plug UNPROVED 
power_at_dist UNPROVED 
power_at_coil UNPROVED 
power.at.poin UNPROVED 
pover_at_igni PROVED 
power_at_batt PROVED 


Totals: 8 proofs, 8 attempted, 2 succeeded. 


Figure 4.7: Result of EHDM ProveModule With comp=ignition 

be observable power through the battery and the ignition. To validate the 
expert system, we simply answer the questions which correspond to a faulty 
ignition switch. A transcript of the results of the expert system run is 
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shown in Figure 4.8. As can be seen from comparing the transcript of the 


Does your car fall to start? (Yss or No) yas 

Does the starter fail to turn when the key is 
engaged? (Yes or No) yes 

Are all of the voltage levels of the individual 
battery cells normal? (Yes or No) yes 

Rule <D1A> 

deduces: (GOOD CELLS) 

Rule <D1> 

deduces : (POWER AT BATTERY) 

Are there any cracks or damage to the battery 
case? (Yes or No) no 

Are the battery postB and cable clamps 
corroded? (Yes or No) no 

Rule <D2A> 

deduces : (GOOD BATTERY) 

Rule <D2> 

deduces : (POWER AT IGNITION) 

Does a voltmeter connected to the starter post 
of the solenoid fall to move when the key is 
turned to the start position? (Yes or No) yes 

Rule <Y6> 

deduces: (BAD IGNITION) 

Diagnosis: BAD IGNITION. 


Figure 4.8: Example Transcript From Expert System 


expert system session with the results of the formal proof, the model and 
the expert system share an inverse relationship. The model was able to 
accurately predict the set of observations consistent with a failed ignition 
switch, and the expert system was able to correctly deduce that the ignition 
switch had failed (while making intermediate deductions consistent with the 
predictions of the model). 
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It is interesting to note that the expert system may not actually observe 
all of the predicted good components on its way to diagnosing a failure. For 
instance, in the case of a failed solenoid, the model’s predictions are shown 
in Figure 4.9. The model predicts that there will be power throughout 

Proof summary for module predict 

pover_at_star UNPROVED 
power_at_sole PROVED 
power_at_plug PROVED 
power_at_dist PROVED 
power_at_coil PROVED 
power_at_poin PROVED 
power.at.igni PROVED 
power_at_batt PROVED 

Totals: 8 proofs, 8 attempted, 7 succeeded. 

Figure 4.9: Result of EHDM ProveModule With comp=solenoid 

all system components with the single exception of the starter. The expert 
system, on the other hand, on its way to finding the correct answer, observed 
only three of the predicted seven good components, as can been seen from 
the transcript included in Figure 4.10. This is illustrative of the notion of 
weaker consistency as explained in Section 2.2.2. In this case, the □ in 

Vc€C,e€ E: P(e,c)D 3e ' : M{c,e') A e' □ e 

corresponds to the relation that the observed effects do not contradict the 
predicted effects and that the predicted effects are a superset of the observed 
effects. 

4.3.5 Observations 

Based on this and other similar experiments, we make the following obser- 
vations and conclusions. 

Development of the initial rules based upon heuristics was a tedious and 
arduous process which left us feeling very unsure about the quality and 
completeness of the rule base for even so small an example as the one under 
consideration here. Quite to the contrary, we found generation of diagnostic 
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rules based upon an explicit representation of the model of system behavior 
to be a relatively straightforward process. Further, because we were working 
from an explicit model, we felt a much higher degree of confidence concern- 
ing the quality of the ensuing rule base. It should be possible to develop 
mechanical translators which are capable of doing much of the transforma- 
tion from the logical notation used in expressing models to a rule-based 
representation, thereby reducing development costs and further increasing 
confidence that errors were not introduced during the transformation pro- 
cess. 

We believe that the notion of model inversion has great potential for 
application to verification of diagnostic expert systems. While the ideas set 
forward here are still in a formative stage, the success experienced to date has 
left us feeling optimistic about the feasibility of developing verification and 
validation techniques that are both effective and well-grounded in theory. 
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Does your car fail to start? (Yss or Ho) yes 

Does the starter fall to turn when the key is 
engaged? (Yes or No) yes 

Are all of the voltage levels of the individual 
battery cells normal? (Yes or No) yes 

Rule <D1A> 

deduces: (GOOD CELLS) 

Rule <D1> 

deduces : (POWER AT BATTERY) 

Are there any cracks or damage to the battery 
case? (Yes or No) no 

Are the battery posts and cable clamps corroded? 

(Yea or No) no 

Rule <D2A> 

deduces: (GOOD BATTERY) 

Rule <D2> 

deduces: (POWER AT IGNITION) 

Does a voltmeter connected to the starter post of 
the solenoid fail to move when the key is turned 
to the start position? (Yes or No) no 

Does the needle of a voltmeter flicker when the key 
is jiggled? (Yes or No) no 

Rule <D3A> 

deduces: (GOOD IGNITION) 

Rule <D3> 

deduces : (POWER AT POINTS) 

Rule <D3> 

deduces : (POWER AT SOLENOID) 

Does the starter buzz or turn the engine slowly when 
a jumper is connected between the battery and starter 
posts of the solenoid? (Yes or No) no 

Does the starter show no response when a jumper is 
connected between the battery and starter posts of 
the solenoid? (Yes or No) yes 

Rule <Y9> 

deduces: (BAD SOLENOID) 

Diagnosis : BAD SOLENOID . 


Figure 4.10: Example TVanseript From Expert System 



Chapter 5 

Conclusions and 
Recommendations 


Formal verification is not a technique to be used in isolation: it should be 
a component of a comprehensive development methodology based on the 
systematic application of formal methods. Due to the cost and difficulty 
of applying such methodologies, they are usually reserved for very critical 
applications, where assured reliability and safety are paramount, and where 
simple and robust designs, conducive to formal analysis, may take prece- 
dence over functionality. For formal verification to be tractable, and its 
results credible, formal analysis must be part of the design process from the 
beginning and must influence the design as it evolves. 

These are not attributes of expert system development and knowledge 
engineering as commonly practiced today. Consequently, formal verification 
has only limited applicability, and possibly even less utility, in contemporary 
expert systems development. This is emphatically not to say that formal 
verification has no part to play in the development of high-quality expert 
systems, but to point out that much more needs to be done besides develop- 
ment of the formal verification techniques themselves. It will be necessary 
to revise much of the practice of expert systems engineering, and some of 
its goals, in order to accommodate requirements for high reliability and 
predictable behavior assured by formal analysis. 

We do not expect — or even advocate — that the field of expert systems 
should confess error and change its working practices and philosophical foun- 
dations. Rather, we suggest the creation of a subdiscipline that will seek 
to develop and apply, from expert systems engineering, the methods and 
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insights that are applicable and beneficial to systems with exacting depend- 
ability requirements. Formal methods and verification should be a signifi- 
cant technical component of that subdiscipline — which should bear a similar 
relationship to its parent field as the engineering of conventional computer 
systems for life-critical applications does to the general field of computer 
and software engineering. 

We believe this report makes three contributions to the development of 
a foundation for the subdiscipline of dependable expert systems: 

• The identification of constraint satisfaction and model inversion as for- 
med specification paradigms for synthetic and analytic expert systems, 
respectively. 

• A formed definition of consistency for rule-based expert systems. 

• Identification of the notion of approximate semantics, and examples 
of how these can be achieved and applied in both declarative and 
imperative forms. 


Recommendations for Further Research 

We suggest that research and development should continue on the following 
three levels of increasing rigor and formality: 

1. The development of formally based analysis and anomaly detection 
tools — such as consistency checkers — that can provide useful, but not 
definitive, help in the construction and quality assurance of conven- 
tional expert systems. 

2. The development of a methodology and took for the creation of de- 
pendable expert systems, based on formal specifications, run-time 
checking, and fault-tolerance techniques. Constraint satisfaction and 
model inversion can provide not only formal specifications for expert 
systems, but executable tests for the satisfaction of those specifica- 
tions. By testing the output of an expert system against its specifi- 
cation at run time, and invoking fault-recovery and tolerance mecha- 
nisms on failure (e.g., the execution of alternate subsystems or algo- 
rithms as in the recovery block method for software fault tolerance), 
it should be possible to develop highly dependable expert systems for 
certain, perhaps limited, applications without requiring a complete 
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change in the development and implementation strategy for the ex- 
pert systems themselves. 

3. The development of a formally-based methodology and tools for ex- 
pert systems. This will require development of a systematic design 
methodology for expert systems in which knowledge engineering is 
combined with the systematic, top-down elaboration of requirements 
and specifications. It will also require development, for expert systems, 
of programming languages that are amenable to formal analysis. One 
slightly radical, but very promising, approach will be to abandon con- 
ventionally hand-crafted, rule-based expert systems for certain critical 
applications, and instead to develop those applications through explicit 
constraint satisfaction and model inversion techniques. The latter may 
involve either the exploration of an explicit model at run time, or the 
“compilation” of that model into an efficient set of production rules. 
Pearce [44] has demonstrated that this technique can be both more 
efficient and more reliable than a hand-crafted rule set. 

We suggest that exploration of the second and third approaches identi- 
fied above should be conducted and evaluated through modest prototyping 
experiments using potential applications of expert systems of interest to 
NASA and Space Station planners. Fault diagnosis (analysis) and schedul- 
ing (synthesis) seem to be the most promising applications, from both the 
scientific and practical points of view. 


48 


Chapter 5. Conclusions and Recommendations 


Appendix A 
P-Best 


The Production-Based Expert System Toolset (P-B&ST) is a forward-chaining, 
LISP-based expert system development environment (sometimes referred to 
as a shell), which consists of a rule-development language, rule compiler, 
run time routines, and a debugger. It provides an integrated set of facilities 
for the creation and debugging of complex knowledge bases. Although de- 
veloped primarily as a research vehicle in the exploration of expert systems, 
its features and facilities have been heavily influenced by the suggestions, 
critiques and demands of individuals involved in the applications of expert 
systems to real-world problems. P-BEST is intended to be portable, yet 
reasonably efficient in executing complex rules against large data sets. 

A.l How P-BEST Works 

The basic P-BEST strategy revolves around maintenance of an expert sys- 
tem state. The system state is represented by the union of the states of the 
fact base, the rule base, and any external data structures unique to a given 
application. This union is referred to as the knowledge base (KB). Rules 
(IF . . . THEN . . . pairs) are developed by the user and compiled by P-BEST 
into the rule base. A very simple rule might say something like: 

if is-a-man socrates, 
then 

is-mortal socrates. 

Once the rules are developed, the user may introduce facts (knowledge) into 
the KB through assertions. For example, a user might wish to assert that 
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is-a-man socrates 

The P-BEST inference engine is a distributed mechanism for matching 
facts to rule antecedent patterns, performing binding analysis, and invoking 
rule consequents. It is distributed in the sense that for each rule in the rule 
base, the compiler will produce a number of primitive LISP functions that 
implement the semantics of the rule. As facts are asserted, each rule in 
the rule base that references the type of knowledge being asserted examines 
the fact for a potential match. When a pattern in the antecedent of a rule 
matches the fact being asserted, a binding is created and associated with 
both the fact and the rule. If such a binding is created, and if, as a result 
of this binding, all the patterns of a rule have been matched, then binding 
analysis is performed to determine if all of the variable values referenced by 
the rule/fact bindings are consistent. 

Given the example rule stated above, if the fact were asserted 

is-a-man socrates 

then the rule would determine that its antecedent had been satisfied. Be- 
cause the sample does not specify any additional constraints, the rule would 
pass consistency analysis and conclude that 

is-mortal socrates 

Of course, if the rule stated something like 

if is-a-man socrates and 
is-alive socrates 
then 

is-mortal socrates 


then asserting that is-a-man socrates would not be sufficient to complete 
the conditions of the rule. 

Most expert systems will consist of a large number of rules and require 
the assertion of a substantial number of facts to reach meaningful conclu- 
sions. Under P-BEST, when a fact is asserted and matched against all 
candidate rules, the resulting set of rules Sagged as complete (all patterns 
in the antecedent are bound to some fact) are examined to determine if the 
there is a valid binding combination. This second level of analysis which 
deals with value constraints is called binding analysis. In general, vari- 
able values are examined and any additional test constraints are evaluated 
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to determine if there is a consistent interpretation of the binding set that 
would allow a rule to reach its conclusion (or fire). The rules with rule/fact 
bindings that meet binding analysis are gathered into a conflict set, and 
the “best” rule in this set is selected and fired. The process of selecting a 
rule to fire from a number of possibilities is called conflict resolution. The 
consequent of the selected rule will fire and generally alter the state of the 
knowledge base, which will cause the creation of other rule/fact bindings. 
The process then repeats until no further candidates for firing remain. It 
is up to the individual application to assign an interpretation to this state: 
in some applications, exhausting all possible inferences might be considered 
the natural culmination of processing, while in other applications, reaching 
such a state might be considered an error. 

A.1.1 Facts 

Facts in the P-BEST system may be thought of as predicates asserted into 
the fact base. At the lowest level, they are LISP structures whose format 
is described by a user-provided definition statement. A fact’s type is called 
its ptype (or pattern-type), and thus the function for defining fact types is 
called “define-ptype.” A sample ptype definition might be: 

(del ine-ptype is-mortal person) . 

This ptype, is-mortal, allows the assertion of facts into the system whose 
person field could contain something like socrates, or reagan or 5. It 
is necessary to note the difference between syntactic constraint, which the 
system enforces, and semantic constraint, which is the responsibility of the 
knowledge engineer. In the above example, is-mortal is a syntactic struc- 
ture that takes a single argument: this is the extent of the restrictions 
enforced by the system. Semantically, the intent of this predicate is that 
the value occupying the person field is a mortal. Asserting, for instance, 
that (is-mortal air) or (is-mortal 5) is syntactically correct, but is of 
dubious semantic worth. 

Although facts are generally asserted into the knowledge base by rules, a 
user can directly alter the state of the knowledge base. The act of asserting a 
fact is handled by the function pbest-assert. The function pbest-assert 
allows the assertion of a single fact into the Fact Base. A fact assertion 
might look like this: 

(pbest-assert ’(is-mortal socrates)) 
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Figure A.l: Knowledge Base 


This would result in the assertion of a fact (or predicate) of type is-mortal, 
whose attribute is socrates. The fact would be added to the global knowl- 
edge base, and appropriate rules would examine the fact in an attempt to 
bind to it. 

A.1.2 Rule/Fact Bindings 

What does it mean for rules and facts to be bound? When a fact is asserted 
into the fact base, all rules pertaining to that fact (i.e., which reference 
facts of a particular p-type) create links to it. These links are referred to as 
bindings. Consider the symbolic examples of Figure A.l, which represents 
the state of the system after three rules (Rx,Ry, R z ) have been added to the 
rule base, and three or more facts (fl, f2, f3,. . .fn) have been asserted to the 
fact base. 

In this example, (where R=rule, P=pattern, and B=binding) rules Rx 
and Ry are candidates for binding analysis to determine if some combination 
of the current bindings constitutes a consistent binding context (i.e., all 
repeated variables have the same value, and all variable tests and restrictions 


i 
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are satisfied). You will notice that R z is not a candidate for binding analysis 
because R t Pzi is not bound to any fact. Therefore, the COMPLETE flag 
(represented by the vertical “C") is not set. Ry, on the other hand, will be 
considered in binding analysis because ~'R y P y 2 , which is a not-pattern, has 
no bindings; all other patterns in Ry have bindings. 

What does it mean for a binding context to be consistent? Briefly, P- 
BEST checks each possible combination of the rule patterns and all matching 
facts until it finds a set of facts that are consistent. So, if a rule has two 
patterns, the first pattern having four facts that match it, and the second 
pattern having five facts that match it, then P-BEST may have to consider 
all possible pairs 1 . In this case, there are potentially 20 different comparisons 
to establish a consistent set of facts. 


A.2 P-BEST Components 

P-BEST consists of a production-rule language, a compiler, and a window 
debugger. These facilities are combined into an integrated environment for 
the creation and maintenance of knowledge-based systems. 

A.2.1 The Language 

The heart of any expert system shell is its facility for capturing knowledge. 
The P-BEST language is very expressive, allowing the knowledge engineer 
to capture complex relationships in a concise form, yet the syntax is con- 
strained English, thereby greatly increasing the readability (and therefore, 
the maintainability) of P-BEST rules over other representations. 

A.2. 2 The Compiler 

The implementation of P-BEST optimizes execution speed and avoids the 
use of an extensive run-time support package in favor of direct translation of 
the production rules into complete, independently compiled functions. Var- 
ious run-time control logic and auxiliary functions are provided to support 
the execution of the rules, but the real heart of a P-BEST-based expert 
system is the generated functions that represent the rules themselves. 

It is perhaps misleading to say that P-BEST contains a compiler; it 
does not. P-BEST contains a translator that converts the rules from its 

1 P- Best’s optimized search strategy allows it to terminate its search upon finding the 
first consistent binding context 
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own representation language to CommonLisp functions. P-BEST then de- 
pends upon the operation of the target system’s CommonLisp compiler to 
complete the process of generating object code. This is all transparent to 
the knowledge engineer, however. 

Each production rule is translated into three main functions that im- 
plement the semantic intent of the rule: a function for examining newly 
asserted facts and creating proper rule/fact bindings; a function for analyz- 
ing binding relationships and determining consistency; and a function that 
implements the results, or consequent, of the rule. 

A.2.3 The Debugger 

The expert system development environment presented by P-BEST is greatly 
enhanced by the addition of a windowing debugger. Although this debugger 
is not intended to be used as put of a completed expert system, it is very 
useful during the creation and debugging of the rule base. 

The debugger provides multiple simultaneous views into the state of 
the knowledge base. In one window, the set of currently asserted facts is 
displayed along with binding information. In another window, the rules 
are displayed. The user is provided with numerous operations, such as 
commands to step through the execution of the rule base, commands to 
alter system state through assertion or negation of facts, and commands to 
activate tracing features. 
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The EHDM verification system was constructed at SRI International under 
the direction of Friedrich von Henke[l, 2, 3, 55]. The following sections 
provide a brief overview of some of the features of the EHDM methodology. 


B.l The EHDM Specification Language and Logic 

This section provides a brief overview of the specification language of EHDM 
and the underlying logic. 

The specification language is based on first-order typed predicate calcu- 
lus, but also includes elements of richer logics, such as higher-order logic [49], 
lambda-calculus [7] and Hoare logic [29], for greater expressiveness. For ex- 
ample, higher-order terms are particularly useful for expressing induction 
schemas and requirements. 

The specification language is strongly-typed; all entities must be declared 
with their type before use. The type system includes subtypes and function 
types. Specifications are written as definitions and formulas (axioms, theo- 
rems, and lemmas). In addition to the standard expressions, the language 
provides the Boolean connectives, polymorphic conditionals, and quantified 
expressions, including quantification over functions . 

A sublanguage is included for modeling operational behavior and imper- 
ative programs, based on the notions of state object and operation. State 
objects correspond to “program variables” in programming languages. Op- 
erations express state transformations; they have an effect on state objects 
by possibly changing their values. 
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The sublanguage also includes constructs for composing operation ex- 
pressions; these correspond to the common control structures of program- 
ming languages. The combination of all these features forms a sublanguage 
that is essentially equivalent to a simple subset of the Ada programming lan- 
guage. The semantics of operations are defined by Hoare formulas, which 
express properties of the states before and after the state transformation 
denoted by the operation. 

Specifications are organized around the concept of parameterized mod- 
ules. Modules are closed scopes with explicit importation and exportation of 
names; modules can be nested. Names are made unique by qualifying them 
with the name of their module of origin. Modules may be parameterized 
by types, constants, and functions. Semantic assumptions or constraints on 
module parameters cam be expressed; these entail an obligation that must 
be justified for each module instantiation. This form of module parameter- 
ization is very general and powerful; it supports generic specifications and 
allows many complex constructs to be built from simple language primitives. 
Modules are the basic building blocks of specifications. A module may rep- 
resent the theory describing a specification concept, an abstract data type, 
an abstract state machine, or an (abstract) program. 

An important aspect of the EHDM language (and the EHDM approach 
in general) is the support of hierarchical development of specifications and 
proofs. The language supports hierarchical structuring of specifications, 
both with respect to composition of modules (“horizontal hierarchy”) and 
levels of abstractions and refinements (“vertical hierarchy”). Vertical links 
between modules at adjoining abstraction levels are established by map- 
pings, which generalize the notion of implementation. 

The language has been designed so that it naturally supports a high de- 
gree of reusability of specifications and proofs. Reusability is also enhanced 
by the library facility described later. 


B.2 The Theorem Prover of EHDM 

The theorem-prover component of EHDM combines powerful heuristics for 
mechanically generating proofs in first-order predicate logic with efficient 
decision procedures for the following standard theories: 

• Ground formulas in propositional calculus 

• Equality over uninterpreted function symbols [50] 
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• Presburger arithmetic, that is, linear arithmetic with the usual order- 
ing relations [51] 

Equational reasoning similar to the use of rewrite rules is specially supported 
by the mechanized proof procedure. The prover also implements the main 
reduction rules of lambda calculus and a fragment of higher-order logic; 
however, the exact extent of this support is currently unclear. 

The system supports both automated proof generation and interactive 
proof construction that depends on user guidance. Proofs (more precisely, 
proof steps) are declared in the proof part of a specification module; they 
are expressed as a conclusion to be proven and a list of formulas (axioms 
and lemmas) from which the conclusion can be deduced. The automated 
prover completes a proof by attempting to construct suitable instances of 
the formulas involved; the user can help in this process by providing some 
substitutions for free variables, either directly in the proof declaration or 
during proof construction. Completed proofs can be captured in augmented 
proof declarations and included in the specification text for later “replay.” 
A proof-chain analysis tool checks for completeness of larger proof trees and 
helps keep track of dependencies. 

A special procedure for reasoning about state transformations has built- 
in knowledge of the meaning of Hoare formulas and the constructs for ex- 
pressing state objects and state transformations. This procedure provides 
the main support for code-level verification. It permits users to reason di- 
rectly with Hoare formulas, without the traditional intermediate step of 
translating annotated programs into verification conditions (VCs); the pro- 
cedure also supports reasoning about program fragments, as opposed to com- 
plete programs units (like subprograms). In these respects, the paradigm 
that is implemented by the procedure is more general than the traditional 
“verification condition generator” (VCG) paradigm. However, the equivar 
lent of a VCG is available in the EHDM environment as a proof development 
tool. 


B.3 The EHDM Environment 

The EHDM environment is implemented as an integrated, interactive system 
that supports all activities involved in creating, analyzing, modifying, man- 
aging, and documenting specification modules and proofs. The standard 
user interface of EHDM uses the bit-map display and combines a display- 
oriented text editor (customized and enhanced EMACS) with multiple win- 
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dows, menus, and mouse input. (A less enhanced editor-based interface is 
available for remote operation.) All operations can be invoked directly from 
the editor, including the basic operations of parsing, prettyprinting, and 
typechecking specification text, invoking the theorem prover, and requesting 
status information, hi addition to the basic operations just mentioned, the 
system provides a number of further support tools, including: the Context 
and Library tools, the configuration control support, and the MLS Checker 
and the EHDM-to-Ada translator (described in [3]). 

B.3.1 The Context and Library Manager 

The system maintains an internal data base for keeping track of the state 
of individual modules and proofs (referred to as the working context ), and 
of the interdependences among modules and libraries; the user can manip- 
ulate this working context or switch between contexts. Modules are the 
basic entities around which the &HDM system is organized, and the con- 
text feature virtually insulates the user from the underlying file system; the 
EHDM system creates and manages ‘internal’ files, which the user can ignore 
completely because all file manipulation happens as a side effect of the user 
interaction with the EHDM system. 

The library mechanism permits users to group together standard mod- 
ules in libraries of reusable concepts, theorems, and proofs. The environ- 
ment offers tools for creating and maintaining module libraries and supports 
sharing of libraries among users and projects. 

Contexts and libraries have been designed so that the novice user can 
completely ignore these facilities. A user always works within a context, but 
when a user starts EHDM for the first time, the system automatically estab- 
lishes a working context; later, when the user leaves EHDM the system saves 
the context and restores it when work is resumed. Users need to know about 
contexts only when they want to work in more than one context; similarly, 
they can ignore libraries until they want to make use of that facility. 

B.3.2 The Configuration Control Tool 

A configuration and version control mechanism ensures that consistent ver- 
sions of modules are used, and status checks report on the status of modules 
and proofs, and on dependencies among modules and proofs. At any given 
time, a module specification may be in one of several states. When a mod- 
ule is typechecked and also when a proof is performed, a “version check” is 
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made to see that the typecheck information recorded for the transitive clo- 
sure of all referenced modules is still valid. For example, if module A uses 
module B, then changes to module B will invalidate the typecheck infor- 
mation recorded for module A. This will be discovered when the typecheck 
information for module A is used during the typechecking of a module that 
uses A, or during the construction of a proof from A or a module that uses 
A. 
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; ; Each position in the 3x3 board is represented in working 
;; memory by a "position” assertion of the following form: 

(define-ptype position row col marker) 

; ; where row, col are integers in the range 1-3 and marker is an 
element of the set free, opp, com. 


A fact is asserted to keep track of whose turn it is 


(define-ptype turn player count) 

;; where player is an element of the set opp, com, and count is 
; ; of type nat . 

; ; The initial state consists of 9 position assertions and 1 turn 
; ; assertion. 


A fact is asserted as the result of a player move, prior to 
altering the board position, so that it may be checked. 
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(del ine-ptype oppjnove row col) 

; ; where row and column are integers in the range of 1-3 which 
; ; must correspond to a free position on the board. 

; ; The following macros are defined for convenience: 

(defmacro comerp (x y) ‘(and (oddp ,x)(oddp ,y))) 

(def macro middlep (x y) '(or (and (oddp ,x)(evenp ,y)) 

(and (evenp ,x)(oddp ,y)))) 
(defmacro centerp (x y) ‘(and (equal ,x 2) (equal ,y 2))) 

• t 
» i 


(defrule blk.cc states 
if there exists a turn called tl 
such that player is com and 
count is 3 and 
there exists a position 
such that row is ?rl and 
col is ?cl and 
marker is opp and 
there exists a position 
such that row is ?r2 and 
col is ?c2 and 
marker is opp 

with (and (not (equal ?rl ?r2)) 
(not (equal ?cl ?c2)) 
(comerp ?ri ?cl) 
(comerp ?r2 ?c2)) 
there exists a position 
such that row is 2 and 
col is 2 and 
marker is com and 



there exists a position called pi 
such that row is ?r3 and 
col is ?c3 and 
marker is free 
with (oiddlep ?r3 ?c3) 
then forget tl and 
forget pi and 
remember a position 

which inherits from pi 

except that marker is com and 
remember a turn 

such that player is opp and 
count is 4) 


(defrule blk_cm states 
if there exists a turn called tl 
such that player is com and 
count is 3 and 
there exists a position 
such that row is ?rl and 

col is ?cl and 

marker is opp and 
there exists a position 

such that row is ?r2 and 

col is ?c2 and 

marker is opp 

with (and (not (equal ?rl ?r2)) 
(not (equal ?cl ?c2)) 
(comerp ?rl ?cl) 
(middlep ?r2 ?c2)) 
there exists a position 
such that row is 2 and 
col is 2 and 
marker is com and 
there exists a position called pi 
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such that row is ?r3 and 
col is ?c3 and 
marker is free 

with (and (and (oddp ?r3) ; ; take corner which intersects 

(oddp ?c3) ) ; ; both of opp ’ s markers 
(or (and (equal ?r3 ?rl) 

(equal ?c3 ?c2)) 

(and (equal ?r3 ?r2) 

(equal ?c3 ?cl)))) 

then forget tl and 
forget pi and 
remember a position 

which inherits from pi 

except that marker is com and 
remember a turn 

such that player is opp and 
count is 4) 


(defrule win.col at rank 1 states 
if there exists a turn called tl 
such that player is com and 
there exists a position 
such that row is ?rl and 
col is ?cl and 
marker is com 
there exists a position 
such that row is ?r2 and 
col is ?c2 and 
marker is com 

with (and (not (equal ?rl ?r2)) 
(equal ?cl ?c2)) and 
there exists a position called pos 
such that col is ?c3 and 
marker is free 
with (equal ?c2 ?c3) 



then 

forget tl and 
forget pos and 
remember a position 

which inherits from pos 

except that marker is com) 


(defrule win_row at rank 1 states 
if there exists a turn called tl 
such that player is com and 
there exists a position 
such that row is ?rl and 

col is ?cl and 

marker is com 
there exists a position 
such that row is ?r2 and 

col is ?c2 and 

marker is com 
with (and (equal ?rl ?r2) 

(not (equal ?cl ?c2))) and 
there exists a position called pos 
such that row is ?r3 and 
marker is free 
with (equal ?r2 ?r3) 
then 

forget pos and 
forget tl and 
remember a position 

which inherits from pos 

except that marker is com) 
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(del rule win_dia at rank 1 states 
if there exists a turn called tl 
such that player is con and 
there exists a position 
such that row is ?rl and 

col is ?ci and 

marker is com 
there exists a position 
such that row is ?r2 and 

col is ?c2 and 

marker is com 

with (and (not (equal ?rl ?r2)) 

(not (equal ?cl ?c2)) 

(equal (abs (- ?rl ?r2)) 

(abs (- ?cl ?c2)))) and 

there exists a position called pos 
such that row is ?r3 and 

col is ?c3 and 

marker is free 

with (and (equal (abs (- ?rl ?r3)) 

(abs (- ?cl ?c3))) 

(equal (abs (- ?r2 ?r3)) 

(abs (- ?c2 ?c3)))) 

then 

forget pos and 
forget tl and 
remember a position 

which inherits from pos 

except that marker is com) 


(defrule blk-col states 

if there exists a turn called tl 
such that player is com and 
count is ?x and 
there exists a position 


i 



such, that row is ?rl and 

col is ?cl and 

marker is opp 
there exists a position 
such that row is ?r2 and 

col is ?c2 and 

marker is opp 

with (and (not (equal ?rl ?r2)) 
(equal ?cl ?c2)) and 
there exists a position called pos 
such that col is ?c3 and 
marker is free 
with (equal ?c2 ?c3) 
then 

forget pos and 
forget tl and 
remember a position 

which inherits from pos 

except that marker is com and 
remember a turn 

such that player is opp and 

count equals (1+ ?x)) 


(defrule blk_row states 
if there exists a turn called tl 
such that player is com and 
count is ?x and 
there exists a position 
such that row is ?rl and 

col is ?cl and 

marker is opp 
there exists a position 
such that row is ?r2 and 

col is ?c2 and 

marker is opp 
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with (and (equal ?rl ?r2) 

(not (equal ?cl ?c2))) and 
there exists a position called pos 
such that row is ?r3 and 
marker is free 
with (equal ?r2 ?r3) 
then 

forget pos and 
forget tl and 
remember a position 

which inherits from pos 

except that marker is com and 
remember a turn 

such that player is opp and 

count equals (1+ ?x)) 


(defrule blk-dia states 
if there exists a turn called tl 
such that player is com and 
count is ?x and 
there exists a position 
such that row is ?rl and 

col is ?cl and 

marker is opp 
there exists a position 
such that row is ?r2 and 

col is ?c2 and 

marker is opp 

with (and (not (equal ?rl ?r2)) 

(not (equal ?cl ?c2)) 

(equal (abs (- ?rl ?r2)) 

(abs (- ?cl ?c2)))) and 

there exists a position called pos 
such that row is ?r3 and 

col is ?c3 and 



marker is free 

with (and (equal (abs (- ?rl ?r3)) 

(abs (- ?cl ?c3))) 

(equal (abs (- ?r2 ?r3)) 

(abs (- ?c2 ?c3)))) 

then 

forget pos and 
forget tl and 
remember a position 

which inherits from pos 

except that marker is com and 
remember a turn 

such that player is opp and 

count equals (1+ ?x)) 


(defrule see_won_game at rank 2 states 
if there exists a turn called tl and 
there exists a position 
such that row is ?rl and 

col is ?cl and 

marker is ?m 

with (not (equal ?m ’free)) and 
there exists a position 
such that row is ?r2 and 

col is ?c2 and 

marker is ?m 

with (or (not (equal ?r2 ?rl)) 

(not (equal ?c2 ?cl))) and 
there exists a position 
such that row is ?r3 and 

col is ?c3 and 

marker is ?m and 

with (and (or (not (equal ?r3 ?r2)) 
(not (equal ?c3 ?c2))) 
(or (not (equal ?r3 ?rl)) 
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(not (equal ?c3 ?cl)))) and 
test (or (and (equal ?rl ?r2) 

(equal ?rl ?r3)) 

(and (equal ?cl ?c2) 

(equal ?cl ?c3)) 

(and (equal (aba (- ?rl ?r3)) 

(abs (- ?cl ?c3))) 

(equal (abs (- ?r2 ?r3)) 

(abs (- ?c2 ?c3))) 

(not (or (equal ?rl ?r2) 

(equal ?cl ?c2))))) 

then forget tl and 

execute (draw-board) and 

execute (format t ”“%Player "A Won The Game!!!”%" ?m)) 


(defrule see-tie-game at rank 2 states 
if there exists a turn called tl 
such that player is ?p and 
there does not exist a position 
such that marker is free 
then forget tl and 

execute (format t M "%Tie Game! !!"%")) 


(defrule make_randomJnove at rank -10 states 
if there exists a turn called tl 
such that player is com and 
count is ?x and 

there exists a position called pi 
such that marker is free 
then forget pi and 
forget tl and 
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remember a position 

which inherits from pi 

except that marker is com and 
remember a turn 

such that player is opp and 

count equals (1+ ?x)) 


(defrule get_opp_move states 
if there exists a turn 

such that player is opp and 
there exists a position 

such that marker is free and 
there does not exist an oppjnove 
then execute (draw-board) and 
remember an oppjnove 

such that row equals (get-ans ’row) and 
such that col equals (get-ans ’col)) 


(defrule do_oppjnove states 

if there exists a turn called tl 
such that player is opp and 
count is ?x and 

there exists an oppjnove called ml 
such that row is ?rl and 

col is ?cl and 

there exists a position called pi 

such that row is ?rl and 

col is ?cl and 

marker is free 
then forget tl and 
forget ml and 


i 
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forget pi and 
remember a position 

which inherits from pi 

except that marker is opp and 
remember a turn 

such that player is com and 

count equals (1+ ?x)) 


(defrule bad_opp_move states 
if there exists a turn 

such that player is opp and 
there exists an opp_move called ml 
such that row is ?rl and 

col is ?cl and 

there exists a position 
such that row is ?rl and 

col is ?cl and 

marker is ?m with (not (equal ?m ’free)) 
then forget ml and 

execute (format t ""%That position is already taken! "%")) 


(defun init-ttt () 


(progn 

(pest -reset) 
(pest -assert 
(pest -assert 
(pest-assert 
(pest-assert 
(pest-assert 
(pest-assert 
(pest-assert 


’(position 3 2 free)) 
’(position 2 3 free)) 
’(position 2 1 free)) 
’(position 1 2 free)) 
’(position 3 3 free)) 
’(position 3 1 free)) 
’(position 1 3 free)) 
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(pest-assert 

(pest-assert 

(pest-assert 


’(position 1 1 free)) 
’(position 2 2 free)) 
’ (turn opp 0) ) ) ) 


(defun get-ans (label) 

(do ((x 0)) 

((and (> x 0)« x 4)) x) 

(format t ”"%Enter "A (1-3) : " label) 
(setf x (read)))) 


(defun draw-board () 

(format t "~%r 
c: 1 : 2 : 3 '%") 

(do ((r l)(c 1 (1+ c))) 

((and (» r 3) (> c 3))) 

(if (> c 3) 

(progn (setf cl) 

(incf r) 

(format t — + — + — “%"))) 

(if (» c 1) (format t " "A : H r)) 

(format t n ~k " (print-marker r c)) 

(if « c 3) (format t "I")))) 


(defun print-marker (row col) 

(dolist (f *facts*) 

(if (and (position-p (fact-body (eval f))) 

(equal (position-row (fact-body (eval f))) row) 
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(equal (position-col (fact-body (eval 1))) col)) 

(if (equal (position-marker (fact-body (eval f))) ’opp) 
(return ’0) 

(if (equal (position-marker (fact-body (eval f))) ’com) 
(return ’X) 

(return #\Space)))))) 
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arrays: Module [dtype: TYPE, undef-r: dtype] 

Exporting array, newarray, assign, *l[*l], length, dtype 

Theory 

array: TYPE 
i,j: VAR nat 
d: VAR dtype 
o: VAR array 
newarray: array 

assign: function [array, nat, dtype — ► array] 

*l[*l]: function [array, nat — ► dtype] 
length: function[array — ► nat] 

acessnew: Axiom newarray[i] = undef_r 

selectassign: Axiom a := *[j] = if t = j then d else a[j] end if 
und-ax: Axiom (* < 0 V * > length(a)) D a[»] = undef_r 
End arrays 



data: Module 

Exporting datum, undef, wild 
Theory 

datum: TYPE IS int 
undef, wild: datum 

daxl: Axiom undef ^ wild 

End data 
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diffs: Module 

Exporting I * - * |,oddp,evenp 

Theory 

n, m: VAR nat 

| * — * |: function [nat, nat — ► nat] 
oddp: function [int — ► bool] 
evenp: function[int —► bool] 

diff_ax: Axiom | n - m |= if n > m then n — m else m — n end if 

oddp-ax: Axiom 

oddp(n) = if n = 1 
then true 

ELSIFn > 1 then oddp(n — 2) else false 
end if 

evenp-ax: Axiom 

evenp(n) = if »» = 0 
then true 

ELSIFn > 1 then evenp(n — 2) else false 
end if 

nat_def_ax: Axiom n > 0 
Proof 

diff Jemma: Lemma | n — m |> 0 

dip: Prove diff Jemma from 

diff_ax {n +— n@cs, m <— m@cs}, 
nat_def_ax {n <— n@CS}, 
nat jdef-ax {n <— m@cs} 

End diffs 
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facts: Module 

Using data, arrays[datum, undef] 

Exporting fact with data, arrays[datum, undef] 
Theory 

fact: TYPE from array 
End facts 
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sets: Module [t: TYPE] 

Exporting set, 0, UNIVERSE, {*1}, €, C, c , n, u, \, *T, 
sets.lesseq, sets. less, sets, times, sets.plus, sets.difference, 
sets.minus, seteq 

Theory 

set: TYPE 
0: set 

UNIVERSE: set 
S1,S2, S3: VAR set 
x, y: VAR t 
{*1}: function[t — * set] 

G: function[t, set — *• bool] 

C: function [set, set — *• bool] 

C: function [set, set — ► bool] 
fl: function[set, set — ► set] 

U: function[set, set — *■ set] 

\: function[set, set — *■ set] 

*T: function [set — *■ set] 
seteq: function[set, set — » bool] 

SETdef: Axiom (x € {x}) A ((y G {x}) ■*> x = y) 

subdef: Axiom (SI C S2) o(Vx:(x€ SI) D (x G S2)) 

psubdef: Axiom 

(SI C S2) (((x G SI) D (x G S2)) A ( 3 y: (y G S2) A -.(y G SI))) 

interdef: Axiom (x G (SI n S2)) O ((x G SI) A (x G S2)) 
uniondef: Axiom (x G (SI U S2)) ■<=>• ((x G SI) V (x G S2)) 
diffdef: Axiom (x G (SI \ S2)) -O- ((x G SI) A -i(x G S2)) 
compdef: Axiom (x G SI) -O- (->(x G SI)) 
eqdef: Axiom Si' = S2 O seteq(Sl, S2) 
eq_r: Formula seteq(Sl, SI) 
eq_s: Formula seteq(Sl, S2) seteq(S2, SI) 
eq_t: Formula (seteq(Sl, S2) A seteq(S2, S3)) D seteq(Sl, S3) 
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sb_t: Formula ((SI C S2) A (S2 C S3)) D (SI C S3) 
seteqdef: Axiom 

seteq(Sl, S2) #(Vx:((iG Si) D (i€ S2)) A ((* e S2) D (x € SI))) 
nulldef: Axiom (* € 0 ) = false 
nulldef2: Axiom (Vx: -i(x € SI)) & SI = 0 
unidef: Axiom (x € UNIVERSE) = true 
sub-reflex: Formula (SI C SI) = true 
inter identity: Formula SI = (SI fi SI) 
inter-commutative: Formula (SI n S2) = (S2 n SI) 
inter-null: Formula (SI n 0 ) = 0 
inter_assoc: Formula (SI D (S2 n S3)) = ((SI D S2) n S3) 
union_conunutative: Formula (SI U S2) = (S2 U SI) 
union-associative: Formula (SI U (S2 U S3)) = ((SI U S2) u S3) 
unionidentity: Formula (SI U SI) = SI 
union-null: Formula (SI U 0 ) = SI 
double_neg: Formula ST = SI 
neg-null: Formula 0 = UNIVERSE 
neg.uni: Formula UNIVERSE = 0 
uni.union: Formula (SI U Si) = UNIVERSE 
nullinter: Formula (SI D Si) = 0 
diffinter: Formula (SI \ S2) = (SI n S2) 

End sets 
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states: Module 
Exporting state, sO, next 
Theory 

state: TYPE IS nat 
sO: state = 0 
si,S2: VAR state 

next: function[state — ► state] = ( A si — *• state : (si + 1 )) 

nonneg: Axiom si > sO 

succ: Axiom si > sO D ( 3 82 : next(s2) = Sj) 


End states 
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ttt_ptypes: Module 

Using facts, states, kbs, sets[fact], diffs 

Exporting row, col, marker, player, count, turn, opp_move, 

position, turn-p, opp_move.p, position^), ttt_ptypes. equals, opp, com, 
free, match Jturn, match-opp .move, match_position, akb, cornerp, 
middlep, centerp 

Theory 

row: nat = 1 
col: nat = 2 
marker: nat = 3 
player: nat = 1 
count: nat = 2 
opp, com, free: datum 

turn, oppmove, position: TYPE from fact 
/: VAR fact 
t,t\: VAR turn 
o: VAR oppmove 

P,Pi,P2,P3,P4,p5,p6,p7,p8,p9: VAR position 

dl,d2,d3: VAR datum 

rx, cx, mx, px: VAR datum 

kbx: VAR kb 

sx: VAR state 

akb: kb 

turn_p: function[fact — ► bool] 

match-turn: function[kb, state, fact, datum, datum — » bool] = 

( A kbx, sx, f, dl, d2— ► bool : 

(/ e kb_inst(kbx, sx)) 

A tum_p(/) 

A (dl = wild V dl = /[player]) A (d2 = wild V d2 = /[count])) 
opp_move_p: function[fact — ► bool] 

match_opp_move: functionjkb, state, fact, datum, datum — ► bool] = 

( A kbx, sx, f, dl, d2— *• bool : 

(/ G kbjnst(kbx,sx)) 

A opp_moveqi(/) 

A (dl = wild V dl = /[row]) A (d2 = wild V d2 = /[col])) 
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position _p: function [fact — ► bool] 

match-position: function[kb, state, fact, datum, datum, datum — ► bool] 

= ( A kbx, sx, f, dl, d2, d3 — * bool : 

(/ € kb Jnst(kbx, sx)) 

A position_p(/) 

A (dl = wild V dl = /[row]) 

A (d2 = wild V d2 = /[col]) A (d3 = wild V d3 = /[marker])) 

equals: function[position, position — ► bool] = 

(Api , P 2 * bool : 
px[row] = p 2 [row] 

A pi [col] = P 2 [col] A pi [marker] = p 2 [marker]) 

cornerp: function[fact — *• bool] = 

( Af— ► bool : position_p(/) A oddp(/[row]) A oddp(/[col])) 
centerp: function[fact — *• bool] = 

( Af— ► bool : position.p(/) A /[row] = 2 A /[col] = 2) 
middlep: function[fact — ► bool] = 

( A f— ► bool : 

position_p(/) 

A ((evenp(/[row]) A oddp(/[col])) 

V (oddp(/[row]) A evenp(/[col])))) 


distinctjconstants: Axiom 
opp ^ com A com ^ free 

A opp 7 ^ free A opp ^ wild A com # wild A free # wild 


tmJth^ax: Axiom length(t) = 2 
oppJth-ax: Axiom length(o) = 2 
pos.lth_ax: Axiom length(p) = 3 

fact-type-1: Axiom turn.p(/) V opp_move_p(/) V position_p(/) 
fact_type_2: Axiom turn_p(/) D -i(opp_move_p(/) V position_p(/)) 
fact_type_3: Axiom opp_move_p(/) D ->(turn_p(/) V position_p(/)) 
fact_type_4: Axiom position_p(/) D ->(opp_move_p(/) v turn_p(/)) 



85 


fact_type_5: Axiom tum_p(t) A -i(tum.p(o) V tum_p(p)) 

fact_type_6: Axiom opp_move_p(o) A -i(opp_move_p(t) V opp_move_p(p)) 

fact_type_7: Axiom position_p(p) A -i(position.p(t) V position^ (o)) 

possible. posit ions: Axiom 
(p € kbJnst(akb,sx)) 

O- (p[rowj > 1 

A p[row] < 3 
Ap[col] > 1 
Ap[col] < 3 

A (p[marker] = free V p[marker] = opp V p[markerj = 

com)) 


duplicate-positions: Axiom 
((p € kb Jnst(akb, sx)) 

A (pi G kb Jnst(akb, sx)) Ap[row] = pi[row] Ap[col] = pi[col]) 
D p [marker] = pi [marker] 


possible-turns: Axiom 
(t € kb_inst(akb, sx)) 

O ((t[player] = com V f [player] = opp) A p[count] > 0) 


End ttt_ptypes 
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ttt_rlsl: Module 

Using ttt-ptypes, states, facts, kbs, sets [fact] 

Exporting make_move, make_moveJhs 

Theory 

/: VAR fact 
r, c: VAR nat 
ti,t2: VAR turn 

Pi>P2,P3,P4,p5,p6,p7,p8,p9,p0: VAR position 
sijSj: VAR state 

make-move: function [kb, state — ► state] 
make.moveJhs: function[kb, state — *• bool] 

make-move_def: Axiom 

( 3 *1 , Pi : 

if match.turn(akb, com, wild) 

A mat ch_position( akb , si , pi , wild, wild, free) 
then make jnove(akb, si) = next(si) 

A ->(ti G kbJnst(akb, next(si))) 

A -i(pi G kb Jnst(akb, next(si))) 

A ( 3 12: 

match .turn (akb, next(si), t2, opp, wild) 

A t2[count] = 1 + 1 \ [count]) 

A (3p2 : 

match_position(akb, next(si), p 2 , wild, wild, com) 
A p 2 [row] = pi [row] A P 2 [col] = pifcol]) 
else make_move(akb, si) = si 
end if ) 

make_moveJhsjdef: Axiom 
make-move Jhs(akb, si) 

O ( 3 ii , pi : 

match-turn (akb, si, fi, com, wild) 

A match_position(akb, si, pi, wild, wild, free)) 


movejorder_l: Axiom 

make_move(akb, si) = next(si) 
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(3pi , P2 • 

(match _position(akb, 2, 2, free) 

D match_position(akb, next(«i),p 2 , 2, 2, com))) 

movejorder.2: Axiom 

make_move(akb, ax) = next(si) 

(3 pi ,P2 ,Ps : 

((-imatch_position(akb, «i, pi, 2, 2, free) 

A match_position(akb, «i,p 2 , 1, l,free)) 

D match_position(akb ) next(si),p 3 , 1, l,com))) 

move_order_3: Axiom 

make Jnove(akb, Si) = next(si) 

( 3 Pi , P2 > P3 , P4 : 

( (-'match _position(akb, si,pi, 2, 2, free) 

A ->match.position(akb, si,P 2 , 1, 1, free) 

A match.position(akb, si, ps, 1, 3, free)) 

D match_position(akb, next(si),p 4 , 1, 3, com))) 

move.order.4: Axiom 

make_move(akb, «i) = next(si) 

(3pi , P 2 > Ps , Pa > p5: 

((->match_position(akb, «i, pi, 2, 2, free) 

A -imatch_position(akb, si,p 2 , 1, l,free) 

A -imatch_position(akb, si, ps, 1, 3, free) 

A match _position(akb, «i, p«, 3, 1, free)) 

D match_position(akb, next(si) , p5, 3, 1 , com))) 

move_order_5: Axiom 

make_move(akb, «i) = next(si) 

O (3pi ,P2 ,P3 ,P4 , p5, p6: 

((->match-position(akb, si,pi, 2, 2, free) 

A -imatch_position(akb, si,p 2 , 1, l,free) 

A -imatch_position(akb, «i, ps, 1, 3, free) 

A -imatch_position(akb, «i,p 4 , 3, 1, free) 

A match_position(akb, «i, p5, 3, 3, free)) 
D match_position(akb, next(si), p6, 3, 3, com))) 
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move_order_6: Axiom 

makejnove(akb, «i) = next (ax) 

(3pi , P 2 , Ps » Pa , p5, p6> p7: 

((-■match _position(akb, si,pi, 2, 2, free) 

A -imatch_position(akb, «i,P 2 , 1, 1, free) 

A -<match_position(akb, s\, p 3 , 1, 3, free) 

A ->match_position(akb, ax,p 4 , 3, 1, free) 

A -imatch_position(akb, si , p5, 3, 3, free) 

A match_position(akb, ax, p6, 1, 2, free)) 

D match_position(akb, next(ai), p7, 1, 2, com))) 

move_order_7: Axiom 

make_move(akb, ai) = next(ai) 

o (3pi , P2 , Pa , Pa , P5, p6, p7, p8: 

((->match_position(akb, ai, pi, 2, 2, free) 

A ->match_position(akb, ai,p 2 , 1, 1, free) 

A ->match_position(akb, ai, ps, 1, 3, free) 

A -<match_position(akb, ax , p\, 3, 1, free) 

A ->match_position(akb, ax, p5, 3, 3, free) 

A -imatch_position(akb, ax, p6, 1,2, free) 
Amatch_position(akb, ax, p7, 2, 1, free)) 
D match_position(akb,next(ax),p8,2, l,com))) 

move .order _8: Axiom 

make_move(akb, ax) = next (aj) 

(3pi , P2 , Ps , P4 , p5, p6, p7, p8, p9: 

((-<match_position(akb, ax,px, 2, 2, free) 

' A -imatch_position(akb, *x, pa, 1, 1, f ree ) 

A — match_position(akb, ax, ps, 1, 3, free) 

A -nnatch_position(akb, ax , Pa, 3, 1, free) 

A — match .position (akb, ax, p5, 3, 3, free) 

A -imatch_position(akb, ax, p6, 1,2, free) 
A-imatch_position(akb, ax, p7, 2, 1, free) 


Amatch_position(akb, ax, p8, 2, 3, free)) 
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Z) match_position(akb, next(a 1 ),p9, 2,3, com))) 

movejorder_9: Axiom 

make_move(akb, ai) = next(ai) 

( 3 Pl , P2 > P3 , P4 , p5, p6, p7, p8, p9, pO: 

((-imatch_position(akb, ai, pi, 2, 2, free) 

A ->match_po8ition(akb, «i,P 2 , 1, 1, free) 

A ->match_position(akb, Si,ps, 1, 3, free) 

A ->match_position(akb, si,p 4 , 3, 1, free) 

A -imatch.position(akb, ai, p5, 3, 3, free) 

A ->match_position(akb, a 1( p6, 1, 2, free) 
A-<match-position(akb, «i, p7, 2, 1, free) 

A-<match-position(akb, ai, p8, 2, 3, free) 

Amatch_position(akb, ai, p9, 3, 2, free)) 

D match_position(akb, next(ai), pO, 3, 2, com))) 


End ttt-rlsl 
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ttt_rls2: Module 

Using ttt_ptypes, states, facts, kbs, sets [fact] 

Exporting win_col, win_row, winjdia, winjcolJhs, win_rowJhs, 
win_dia_lhs 

Theory 

/: VAR fact 
t,ti,t2: VAR turn 
p,cl,c2,pi,p2>P3>P4 : VAR position 
8 i,S 2 : VAR state 

winjcol: function[kb, state — * state] 
winjrow: function[kb, state — ► state] 
win_dia: functionfkb, state — » state] 
winjcolJhs: functionfkb, state — ► bool] 
win-row _lhs: functionfkb, state — ► bool] 
win-diaJhs: functionfkb, state — ♦ bool] 

winxol-def: Axiom 
( 3 h , pi , p 2 , pa : 

if match_turn(akb, $i,t\, com, wild) 

A match_position(akb, si , pi , wild, wild, com) 

A match_position(akb, «i, P 2 , wild, wild, com) 

A match_position(akb, si, ps, wild, wild, free) 

A pi [col] =p 2 [col] 

A pi [row] P 2 [row] 

A Ps[col] = p 2 [col] 

A ps[row] ^ pi [row] A p 3 [row] ^ P 2 [row] 

then ( 3 p 4 : 

win_col(akb, si) = next(si) 

A — i(ii € kb-inst(akb,next(si))) 

A -i(ps € kb_inst(akb, next(si))) 

A match_position(akb, next(si) , p<, wild, wild, com) 
A p 4 [row] = pa [row] 

A p 4 [col] = p 3 [col] 

A (Vp: 

P # P 3 

D ((p € kb_inst(akb, si)) 

& (p € kbinst(akb,next(si)))))) 



else win_col(akb, si) = «i 
end if ) 
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win-row jdef: Axiom 
( 3 *x , pi ,P2 , ps : 

if match_tum(akb ) ai,ii, com, wild) 

A match _position(akb, si , pi , wild, wild, com) 

A match_position(akb, «i, p2, wild, wild, com) 

A match _position(akb, *i, ps, wild, wild, free) 

A pi [col] #p 2 [col] 

A pi [row] = p2 [row] 

A p2[row] = ps[row] 

A p 3 [col] # pi [col] A pa [col] # p2 [col] 

then ( B P4 : 

win.row(akb, «i) = next(si) 

A — >(ii € kbJnst(akb,next(«i))) 

A — '(ps 6 kbJnst(akb,next(«i))) 

A match_position(akb, next («i ), p4, wild, wild, com) 
A p4[row] = ps[row] 

A p 4 [col] = ps[col] 

A ( Vp: 

P^P 8 

D ((p € kb_inst(akb, «i)) 

(p € kb_inst(akb,next(«i)))))) 

else win_row(akb, si) = si 
end if ) 

win_diajdef: Axiom 

(3ti ,pi ,p 2 ,p S - 

if match_turn(akb, si,ti, com, wild) 

A match_position(akb, «i, pi, wild, wild, com) 

A match_position(akb, «i, P2, wild, wild, com) 

A match_position(akb, «i, pj, wild, wild, free) 

A pi [row] 7^ p2 [row] 

Api[colj #p 2 [col] 

A pi [row] 7^ ps [row] 

Api[colj T^psfcol] 

A p2[row] # psfrow] 
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A P2 [col] # P3[col] 

A ((cornerp(pi) 

A centerp(pj) A centerp(ps)) 

V (centerp(pi) 

A cornerp(p2) A cornerp(p3))) 


then ( 3 P4 : 

win_dia(akb, si) = next(si) 

A -i(ti € kbjnst(akb, next(sx))) 

A ->(p3 G kbJnst(akb,next(si))) 

A match_position(akb, next(si), P4, wild, wild, com) 
A p4[row] = ps[row] 

A p 4 [col] = p 3 [col] 

A ( Vp: 


P^Pz 

D (( p G kb unst(akb, si)) 

^(pG kbJnst(akb, next(si)))))) 

else winjdia(akb, si) = si 
end if ) 


win_colJhs_def: Axiom 
win_colJhs(akb, si) 

O ( 3 ti , Pi , P2 , Ps : 

match _tum(akb, sj, ti, com, wild) 

A match-position(akb, «i,Pi, wild, wild, com) 

A match_position(akb, si , P2, wild, wild, com) 

A Pi [col] = P2 [col] 

A pi [row] ^ p2[row] 

A match_position(akb, si, ps, wild, wild, free) 
A Ps[col] = P2[col]) 


win_rowJhs_def: Axiom 
win jow Jhs(akb, si) 

O ( 3 *i . Pi > Pi i Ps : 

match_turn(akb, si, ti, com, wild) 

A match_position(akb, «i,pi, wild, wild, com) 

A match.position(akb, »i, P2, wild, wild, com) 

A match_position(akb, si , P3, wild, wild, free) 
A Pi [col] 7* P2[col] 



A pi [row] = p2[row] A p2[row] = ps[row|) 


win_diaJhs_def: Axiom 
win_dia_lhs(akb, «i) 

& ( 3 *i , Pi , P 2 , Ps : 

match_turn(akb, si, *i, com, wild) 

A match_position(akb, 3 i,Pi, wild, wild, com) 

A match_position(akb, s \ , P2 , wild, wild, com) 

A match_position(akb, «i, ps, wild, wild, free) 

A pi [row] ^ p2[row] 

Api[colj #p 2 [col] 

A pi [row] ^ psfrow] 

A pi [col] ^ P 3 [col] 

A p2[row] ^ P3[row] 

A p 2 [col] # P3[col] 

A ((cornerp(pi) 

A centerp(p2) A centerp(ps)) 

V (centerp(pi) 

A cornerp(p2) A cornerp(p3)))) 


End ttt_rls 2 
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ttt_rls3: Module 

Using ttt_ptypes, states, facts, kbs, sets [fact], difFs 

Exporting blk_col, blk_row, blk_dia, blkjcoUhs, blk _row_lhs, 
blkjdiaJbs 

Theory 

/: VAR fact 
*i,t2: VAR turn 

cl,c2,pi,p2,ps,P4: VAR position 
61 , 82 : VAR state 

blkxol: function[kb, state — ► state] 
blkjrow: function [kb, state — *• state] 
blkjdia: function[kb, state — ► state] 
blkxolJhs: function[kb, state — ► bool] 
blk_row_lhs: function [kb, state — *■ bool] 
blk_diaJhs: functionfkb, state — ► bool] 

blkxoLdef: Axiom 
( 3 ti , pi , P2 , P3 : 

if match_turn(akb, si , ti , com, wild) 

A match_position(akb, Si, pi, wild, wild, opp) 

A match_position(akb, sj, P2, wild, wild, opp) 

A pi [col] =p 2 [col] 

A pi [row] 7^ P2[row] 

A match_position(akb, si, ps, wild, wild, free) 

A P2 [col] = ps [col] 
then ( 3 P4 , t2: 

blk_col(akb, Si) = next(si) 

A-«(ti € kbJnst(akb,next(si))) 

' A ->(p3 6 kb Jnst(akb, next(si))) 

Amatch_position(akb, next($i ) , P4, ps [row] , ps[coI] , com) 

A match_turn(akb, next(si), t2, opp, wild) 

A t2 [count] = 1 + ti [count]) 
else blk_col(akb, si) = «i 
end if ) 

blk_row_def: Axiom 
( 3 ti , Pi , P2 , Ps : 
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if match_turn(akb, di,£i, com, wild) 

A match_position(akb, «i, pi, wild, wild, opp) 

A match_position(akb, «i, p 2 , wild, wild, opp) 

A pi [row] = p 2 [row] 

A pi [col] #p 2 [col] 

A match_position(akb, si , ps , wild, wild, free) 

A P2 [row] = P3 [row] 
then ( 3 P4 , t2: 

blk_row(akb, «i) = next(si) 

A — «(ii E kbJnst(akb, next(si))) 

A -i(ps € kb jnst(akb,next(si))) 

Amatch_position(akb, next(«i) , p<, psfrow] , ps[col] , com) 

A match_turn(akb, next(si), t2, opp, wild) 

A 1 2 [count] = 1 + ti [count]) 
else blk_row(akb, «i) = $i 
end if ) 

blk_dia_def: Axiom 

( 3 *1 > Pi > Pi , Ps ■ 

if match_turn(akb,si,ti, com, wild) 

A match_position(akb, «i,pi, wild, wild, opp) 

A match_position(akb, $i,p 2 , wild, wild, opp) 

A match_position(akb, Si, P3, wild, wild, free) 

A pi [row] ^ p2 [row] 

A Pi [col] 9^ p 2 [col] 

A | pi [row] - p 2 [row] |=| pi[col] - p 2 [col] | 

A | pi [row] - p 3 [row] |=| pi[col] - ps[col] | 

A | p 2 [row] - p 3 [row] |=| p2[col] - ps[col] | 

then ( 3 p4 , t2: 

blk_dia(akb, sj) = next(si) 

A — i(ii E kbJnst(akb, s 2 )) 

A ->(p3 E kb_inst(akb, s 2 )) 

A match_position(akb, s 2 > P4> wild, wild, com) 

A p4[row] = psfrow] 

A p 4 [col] =p 3 [col] 

A match_turn(akb, a 2 , t2, opp, wild) 

A t2[count] = 1 + 1\ [count]) 
else blk_dia(akb, «i) = si 
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end if ) 

blk_col_lhs_def: Axiom 
blk_col Jhs(akb, si) 

( 3 , Pi , P2 , Ps : 

match_turn(akb, «i, t\ , com, wild) 

A match_position(akb, si, pi, wild, wild, opp) 

A match_position(akb, si,p 2 , wild, pi[col], opp) 

A pi [row] ^ p 2 [row] 

A match_position(akb, si,p 3) wild, p 2 [col], free)) 

blkjrowJhs-def: Axiom 
blk-row Jhs(akb, si) 

( 3 ii , pi , P2 , Ps : 
match_turn(akb, «i,ti, com, wild) 

A match_position(akb, ei, pi, wild, wild, opp) 

A match_position(akb, $i , P 2 , pi [row] , wild, opp) 

A pi [col] #p 2 [col] 

A match. position(akb, ei , p 3 , p 2 [row] , wild, free) ) 

blkjdiaJhs_def: Axiom 
blk-dia_lhs(akb, «i) 

O ( 3 *i , pi , p 2 , Ps : 

match_turn(akb, «i, ti, com, wild) 

A match.position(akb, ei,pi, wild, wild, opp) 

A match_position(akb, «i , p 2 , Pi [row] , wild, opp) 

A match_position(akb , «i,ps,p 2 [row] , wild, free) 

A pi [row] ± p 2 [row] 

A pi [colj ^p 2 [col] 

A [ pi [row] - p 2 [row] |=| pi[col] - p 2 [col] | 

A | pi [row] - p 3 [row] |=| pi [col] - p 3 [col] | 

A | p 2 [row] - p 3 [row] |=| p 2 [col] - p 3 [col] |) 


End ttt _rls3 



ttt_rls 4 : Module 

Using ttt_ptypes, states, facts, kbs, sets [fact] 

Exporting blkxc, blkxm, blkxcJhs, blkxmJhs 

Theory 

/: VAR fact 
r, c: VAR nat 
ti,t2: VAR turn 

P, PX,Pi,P2,P3,P4,p5: VAR position 
a l>®2 : VAR state 
blkxc: function[kb, state — *• state] 
blkxm: function[kb, state — ♦ state] 
blkxcJhs: function[kb, state — *■ bool] 
blkxmJhs: function[kb, state — ► bool] 

blkxcjdef: Axiom 

( 3 h , pi , p 2 , ps , pi : 

if match.turn(akb, si,ti,com,3) 

A match_position(akb, si,pi, wild, wild, opp) 

A match.position(akb, si, P2, wild, wild, opp) 

A match_position(akb, Si, pa, 2, 2, com) 

A match_position(akb, si, P4, wild, wild, free) 

A pi [row] 7^ P 2 [row] 

A Pi [col] #P2 [col] 

A cornerp(pi) 

A cornerp(p2) 

A centerp(ps) 

A middlep(p4) 

A ( V px: 

((px € kbJnst(akb, si)) 

A px [marker] ^ free) 

O (P x = Pi 
V px = p 2 V px = p 3 )) 

then blkxc(akb, si) = next(si) 

A -i(ti e kb Jnst(akb, next(«i))) 

A -i(p4 G kb Jnst(akb, next(si))) 

A ( 3 12 : 

match_turn(akb, next(si), t2, opp, wild) 
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A t2[count] = 1 + ti [count]) 

A ( 3 p5: 

match_position(akb, next(si) , p5, wild, wild, com) 

A p5[row] = P4[row] 

A p5[col] = P4 [col] 

A ( Vp: 

(p#P4 Ap# p5) 

<=► ((p € kb_inst(akb, si)) 

<^(p6 kb_inst(akb,next(si)))))) 


else blk.cc(akb, «i) = «i 
end if ) 

blk_cm_def: Axiom 

( 3 h , Pi , P2 , Ps , P4 : 

if match_turn(akb,«i, ti.com, 3) 

A match_position(akb, si,pi, wild, wild, opp) 

A match_position(akb, si,P2, wild, wild, opp) 

A match_position(akb, si,p3, 2, 2, com) 

A match_position(akb, sj , P4, wild, wild, free) 

A pi [row] 7^ p2 [row] 

A pi [col] ^ p 2 [col] 

A comerp(pi) 

A middlep(p2) 

A cornerp(p4) 

A (Wrow] = pi [row] A p 4 [col] = p 2 [col]) 
V (p4[row] = p2 [row] 

Ap 4 [col] =pi[col])) 
then blkjcm(akb, si) = next(si) 

A -, (ti € kb jnst(akb, next(si))) 

A ->(p4 € kb_inst(akb,next(si))) 

A ( 3 12: 

match_turn(akb, next(si), t2, opp, wild) 

A 1 2 [count] = 1 + ti [count]) 

A ( 3 p5: 

match_position(akb, next(si), p5, wild, wild, com) 
A p5[row] = p4[row] 

A p5[col] = P4 [col] 

A(Vp: 



99 


P^P< 

■& (p € kb jnst(akb, Sj)) D (p G 

kbJinst(akb, next(«i))))) 

else blkjcm(akb, «i) = «i 
end if ) 


blk_cc Jh s_def: Axiom 
blk_cc_lhs(akb, Si) 

O ( 3 h , Pi , p 2 , P3 , P4 : 

match_turn(akb, sj, £ x , com, 3) 

A match _position( akb , si, pi, wild, wild, opp) 

A match_position(akb, Si,p 2 , wild, wild, opp) 

A match _position(akb, si, p 3 , 2, 2, com) 

A match_position(akb, si, P 4 , wild, wild, free) 

A pi [row] pi [row] 

A pi [col] 7 ^ p 2 [col] A cornerp(pi) A comerp(p 2 ) A 

middlep(p 4 )) 


blk-cmJhs-def: Axiom 
blk_cc_lhs(akb, «i) 

& ( 3 ti , pi , p 2 , P3 > P4 '• 

match_turn(akb, «i, <i, com, 3) 

A match_position(akb, «i, pi, wild, wild, opp) 

A match.position(akb, s \ , P 2 , wild, wild, opp) 

A match_position(akb, si, p 3 , 2, 2, com) 

A match_position(akb, si, P4, wild, wild, free) 

A pi [row] ^ p 2 [row] 

A pi [col] #p 2 [col] 

A cornerp(pi) 

A middlep(p 2 ) 

A cornerp(p 4 ) 

A ((p 4 [row] = pi [row] Ap 4 [col] = p 2 [col]) 
V (p4[row] = p2[row] A P4[col] — 

piM)))) 


End ttt_rls4 
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guardecLops: Module 

Using ttt-ptypes, ttt_rlsl, ttt_rls2, ttt_rls3, ttt_rls4, states, kbs 

Exporting tttjop, g_winjcol, g_win_row, g_win_dia, g_blk.cc, 
gJblkjcm, g-blk jcoI, g_blk_row, g_blk_dia, g_mke_mve 

Theory 

tttjop: TYPE from function [kb, state — ► state] 

g_winjcol: tttjop 

g_win_row: tttjop 

g_win_dia: tttjop 

g_blk_cc: tttjop 

g_blk_cm: tttjop 

g.blk.col: tttjop 

g_blk_row: tttjop 

g.blk.dia: tttjop 

g_mke_mve: tttjop 

gop: VAR tttjop 

8 n : VAR state 

g_winjcol_def: Axiom 
g_win_col(akb, s„) 

= if win_col Jhs(akb, s n ) then win_col(akb, s n ) else s n end if 


g_win_rowjdef: Axiom 
g_win_row(akb, s n ) 

= if ((->win_col Jhs(akb, s n )) A win_row_lhs(akb, s n )) 
then win_row(akb, s n ) 
else 8 n 
end if 


g.winjdiajdef: Axiom 
g_winjdia(akb, s n ) 

= if ((--(winjcol Jhs(akb, s„) V win_row_lhs(akb, s„))) 
A win_dia_lhs(akb, s n )) 
then win_dia(akb, s n ) 
else a n 
end if 
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g_bik-CC-def: Axiom 
g_blk.cc(akb, «„) 

= if ((-i(winjcolJhs(akb, s„) 

V win_rowJhs(akb, s n ) V win_dia_lhs(akb, s n ))) 
A blkjcc Jhs(akb, s n )) 

then blkjcc(akb, «„) 
else s n 
end if 

g_blkjcmjlef: Axiom 
g_blk_cm(akb, s n ) 

— if ((-’(win.col Jhs(akb, s n ) 

V win_row _lhs(akb, s„) 

V win.diaJhs(akb, s n ) V blk_ccJhs(akb, s n ))) 
A blkjCoUhs(akb, s„)) 
then blk_col(akb, s n ) 
else s„ 
end if 


g_blk_coLdef: Axiom 
g_blk_col(akb, s n ) 

= if ((-i(win_colihs(akb, s n ) 

V win_row Jhs(akb, s„) 

V win_diaJhs(akb, s n ) 

V blkjcc Jhs(akb, s n ) V blk jcm Jhs(akb, s n ))) 
A blk_coUhs(akb, s„)) 
then blk_col(akb, s n ) 

else s„ 
end if 


g_blk_rowjdef: Axiom 
g_blk_row(akb, s n ) 

= if ((-i(win_colJhs(akb, s„) 

V win jow Jhs(akb, s n ) 

V winjdia Jhs(akb, s n ) 

V blk_cc Jhs(akb, s„) 

Vblk_cmJhs(akb, s n )vblk_colJbs(akb, «„))) 
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A blk_row_lhs(akb, s n )) 
then blk_row(akb, e n ) 
else s n 
end if 

g-blk-dia_def: Axiom 
g_blkjdia(akb, s n ) 

= if ((->(winxol_lhs(akb, « n ) 

V win_row _lhs(akb, s n ) 

V win-dia Jhs(akb, s n ) 

V blk_cc Jhs(akb, a n ) 

V blk_cmJhs(akb, s„) 

Vblk_col Jhs(akb, « n ) Vblk_row Jhs(akb, s n ))) 

A blk_dialhs(akb, a n )) 
then blk_dia(akb, s n ) 
else 8 n 
end if 


g_mke_mve_def: Axiom 
g_mke_mve(akb, s n ) 

= if ((-i(wirucol _lhs(akb, s n ) 

V win_rowJhs(akb, s„) 

V win_diaJhs(akb, s„) 

V blk.cc Jhs(akb, s n ) 

V blk_cmJhs(akb, s n ) 

V blk_col_lhs(akb, 8 n ) 

vblk_rowJhs(akb, s n )vblk_diaJhs(akb, s n ))) 

A make_move_lhs(akb, a n )) 
then make _move(akb, s n ) 
else s n 
end if 


gop-def: Axiom 
gop = g.win.col 
V gop = g.win_row 
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V gop = g-winjdia 
V gop = gJblkjcc 
V gop = gJblk-cm 
V gop = gJblkjcol 

V gop = g-blk _row V gop = g_blk_dia V gop = g_mke_mve 


gmm_ax: Axiom true 
End guaxded-ops 
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safe.ttt: Module 

Using ttt_ptypes, kbs, states, facts, difis 

Exporting safe, won .game, lose _next_move, unsafe_config_l, 
unsafe_config_2, unsafe_config_3, unsafe.config_4 

Theory 

s n ,sm: VAR state 
ti,t2: VAR turn 
Pi,P 2 ,Ps,P 4 - VAR position 
kbx: VAR kb 

won-game: function[kb, state — + bool] 
lose-next .move: function[kb, state — ► bool] 
unsafe .config-l: function[kb, state — ♦ bool] 
unsafe_config_2: function [kb, state — ► bool] 
unsafe_config_3: function[kb, state — * bool] 
unsafe_config_4: function [kb, state — ► bool] 
safe: function[kb, state — ► bool] 

safe-def: Axiom 
safe(kbx, s„) 

<$■ won_game(kbx, s n ) 

V (lose .next .move (kbx , s n ) 

V unsafe_config_l(kbx, a n ) 

V unsafe_config_2(kbx, a n ) 

V unsafe _config_3 (kbx, a„) V unsafe_config_4(kbx, s n )) 


won-game-def: Axiom 
won_game(kbx, s n ) 

& ( 3 Pi > P2 > Pz • 

match.position(kbx, s n , pi, wild, wild, com) 

A match_position(kbx, s„,p2, wild, wild, com) 

A match .position (kbx, s n ,ps, wild, wild, com) 

A Pi 5 £ P2 

Aptjzpz 

A Pi ± P3 

A ((pi[col] = P2[col] A p 2 [col] = ps[col]) 

V (pi[row] = P2[row] A p 2 [row] = p 3 [row]) 
V (cornerp(pi) 
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A cornerp(j>2) 

A centerp(ps) 

Api [row] ^ p2[row] Api[col] ± 

PaM))) 


lose_next_movejdef: Axiom 
lose_next-move(kbx, a n ) 

& (3*i ,Pi ,P2 ,Ps : 

match_turn(kbx, s n , t \ , opp, wild) 

A match_position(kbx, s„,pi, wild, wild, opp) 

A match.position(kbx, a n ,p2, wild, wild, opp) 

A match_position(kbx, 8 n ,pz, wild, wild, free) 

A Pi # P2 

A ((cornerp(pi) 

A cornerp(p2) 

A centerp(ps) 

A pi [row] 7^ p2[row] 

A pi [col] 7^ P2 [ c °l] ) 

V (pi[row] = p2[row] Ap2[row] = ps[row]) 
V (pi [col] = P2 [col] A P2 [col] = P3[col]))) 


unsafe.config_l.def: Axiom 
unsafe-config J.(kbx, a„) 

•<=>•( 3 *i , pi ,P 2 , pz , Pa ' 

match_turn(kbx, s n , *i, opp, 4) 

A match.position(kbx, a n , px, wild, wild, com) 

A match_position(kbx, a n , p2, wild, wild, com) 

A match _position(kbx, s n , ps, wild, wild, opp) 

A match.position(kbx, a n , p+, wild, wild, opp) 

A pi t* P2 

A ps 7^ P4 

Acenterp (pi ) Acorner p (p2 ) Acornerp (pz ) Acornerp (p* ) ) 


unsafe_config_2jdef: Axiom 
unsafe _config_2(kbx, s n ) 

O ( 3 *i > Pi , P2 , P3 , P4 : 

match_turn(kbx, s n , ti , opp, 4) 
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A match_position(kbx, s n , pi, wild, wild, com) 

A match_position(kbx, a n , P2> wild, wild, com) 

A match _position(kbx , a n , p 3 , wild, wild, opp) 

A match_position(kbx, s n ,P4, wild, wild, opp) 

Api t*P 2 
A ps # P4 

Amiddlep(pi)Amiddlep(p 2 )Acornerp(p 3 )Acornerp(p 4 )) 


unsafe_config_3_def: Axiom 
unsafe_config_3(kbx, s n ) 

O (3t! ,p! ,p 2 ,p s ,p 4 : 

match_turn(kbx, s n , 1 1 , opp, 4) 

A match_position(kbx, a„, pi, wild, wild, com) 

A match_position(kbx, s n , p 2 , wild, wild, com) 

A match_position(kbx, * n , p 3 , wild, wild, opp) 

A match_position(kbx, s n ,p 4 , wild, wild, opp) 

A pi ^ P2 
A ps ^ p 4 

Acornerp(pi)Acornerp(p 2 )Amiddlep(ps)Amiddlep(p 4 )) 


unsafe.config_4.def: Axiom 
unsafe.config.4(kbx, s n ) 

•O' ( 3 ii , pi , P 2 , Ps , P4 : 

match.turn(kbx, s n , ti, opp, 4) 

A match_position(kbx, s n ,pi, wild, wild, com) 

A match.position(kbx, s n , p 2 , wild, wild, com) 

A match_position(kbx, s n ,p 3 , wild, wild, opp) 

A match_position(kbx, s n , p 4 , wild, wild, opp) 

A pi # P2 
A ps ? P4 
A centerp(pi) 

A cornerp(p 2 ) 

A cornerp(ps) 

A middlep(p 4 ) 

A-i((p 2 [rowj = p 3 [row]Ap 2 [col] = P4[col|) 

V (p 2 [row] = p 4 [row] 

A P 2 [col] = P3 [col] ) ) ) 
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End safe-ttt 
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initial _state: Module 

Using ttt_ptypes, kbs, states, facts, sets [fact] 

Theory 

p,pi: VAR position 
i,ti: VAR turn 
sx: VAR state 

initial -positions: Axiom (p € kb _inst(akb, s0)) -o- p[marker] = free 
initial-turn: Axiom 

(t € kb inst(akb, s0)) O t [player] = opp A t [count] = 0 
End initial .state 



initial -proof: Module 

Using ttt_ptypes, safe.ttt, states, kbs, initial .state 
Theory 

sis: Lemma safe(akb, sO) 

Proof 

p: VAR position 

rx, cx, mx: VAR datum 

next_move: Lemma ->lose_next_move(akb, sO) 

nmp: Prove next-move from 

lose .next _move jdef {kbx ♦— akb, s n <— sO}, 
initial-positions {p <— pi@pls}, 
distinct jconstants 

configl: Lemma -iunsafe_config_l(akb, sO) 

clp: Prove configl from 

unsafe.configJ.-def {kbx <— akb, s n <— sO}, 
initial-positions {p *— pi@pls}, 
distinct .constants 

config2: Lemma -<unsafe_config-2(akb, sO) 

c2p: Prove config2 from 

unsafe-config-2_def {kbx *— akb, a n «— sO}, 
initial-positions {p <— pi@pls}, 
distinct-constants 

config3: Lemma -<unsafe_config-3(akb, sO) 

c3p: Prove config3 from 

unsafe.config-3-def {kbx *— akb, s n «— sO}, 
initial-positions {p <— pi@pls}, 
distinct-constants 

config4: Lemma -iunsafe_config_4(akb, sO) 

c4p: Prove config4 from 

unsafe.config-4-def {kbx ♦— akb, s n <— sO), 
initial-positions {p pi@pls}, 
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distinct .constants 

sisp: Prove sis from 

safe-def {kbx «— akb, s n <— s0}, 

next_move, 

configl, 

config2, 

config3, 

config4 

End initial-proof 
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gwr_proof: Module 

Using guardecLops, safe.ttt, ttt_ptypes, states, ttt_rls2 

Theory 

s n : VAR state 

safe_gwr_step: Lemma safe(akb, s n ) D safe(akb, g_win_row(akb, s n )) 

Proof 

no-transition: Lemma 

(safe(akb, s n )Ag_win_row(akb, s n ) = s n ) D safe(akb, g_win_row(akb, s„)) 


ntp: Prove no-transition 
safe-trans: Lemma 

(safe(akb, s n ) A g_win_row(akb, s n ) = next(s„)) 

D safe(akb, g_win_row(akb, s n )) 

stll: Lemma 

g_win_row(akb, s n ) = next(s n ) 

D (win_row Jhs(akb, s n ) A winjrow(akb, a n ) = next(a„)) 

stllp: Prove stll from g_win_rowjdef {s n <— s„@cs} 

stl2: Lemma win_row(akb, s n ) = next(s„) D won-game(akb,win_row(akb,s n )) 


stl2p: Prove stl2 from 

win jow_def {si <— s n @cs, p <— pi@pls}, 
win_row_def {«i <— s n @cs, p <— p 2 @pls}, 
won_game_def {kbx «— akb, 
s n *— next(s n @cs), 

Pi ♦“ Pi@pls, 
p 2 «- P2@pls, 

P3 «- p 4 @pls}, 
distinct-constants 

stpp: Prove safe.trans from 
stll {s n «— s n @CS}, 
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stl2 {e n <- s„@CS}, 

safe_def {kbx «— akb, a n «— win_row(akb, a n @cs)} 

sgsll: Lemma wuurow(akb, s n ) = s n V win_row(akb, s n ) — next(s n ) 

sgsllp: Prove sgsll from win_row_def {si «— s n @cs} 

safe .gwr .step .proof: Prove safe_gwr .step from 
no.transition {«„ «— s n @cs}, 
safe_trans {s n «— s n @cs}, 
sgsll {s„ <- s n @cs}, 
g.winjrowjdef {s n «— s n @cs} 

End gwr.proof 
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gwc-proof: Module 

Using guardedjops, safe.ttt, ttt_ptypes, states, ttt_rls2 

Theory 

s n : VAR state 

safe-gwcjstep: Lemma safe(akb, s n ) D safe(akb, g_win_col(akb,s n )) 

Proof 

no-transition: Lemma 

(safe(akb, s„) A g_win_col(akb, s n ) = s n ) D safe(akb, g_win_col(akb, s n )) 


ntp: Prove no-transition 
safe.trans: Lemma 

(safe(akb, s n ) A g_win_col(akb, s n ) = next(s„)) 

D safe(akb, g_win_col(akb, s„)) 

stll: Lemma 

g_win_col(akb, s n ) = next(s n ) 

D (win_colJhs(akb, s n ) A win_col(akb, s n ) = next(s n )) 

stllp: Prove stll from g_win_col_def {a n *— s n @cs) 

stl2: Lemma win_col(akb, a n ) = next(s n ) D won-game(akb, winxol(akb, a n )) 

stl2p: Prove stl2 from 

winjcoljdef {«i *— s n @cs, p <— pi@pls}, 
winjcoljdef {«i «— s n @cs, p *— p 2 @pls}, 
won_game_def {kbx <— akb, 
s n <— next(s n @cs), 

Pi Pi@pls, 

P2 *- P2@pls, 

Ps «- P4@pls}, 
distinct jconstants 

stpp: Prove safe_trans from 
stll {s n <— s n @CS}, 
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stl2 {a„ «— 3„@CS}, 

safejdef {kbx <— akb, a„ <— winjcol(akb, s n @cs)} 

sgsll: Lemma winjcol(akb, s„) = s n V win_col(akb, s n ) = next(s n ) 

sgsllp: Prove sgsll from winjcol-def {si <— s n @cs} 

safe_gwcjstep_proof: Prove safe-gwc-step from 
no-transition {s n «— s n @cs}, 
safe.trans {s n <— s n @cs}, 
sgsll {s„ <- s„@cs}, 
g_win_col.def {s n ♦— s n @cs} 

Cud gwc .proof 
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gwd_proof: Module 

Using guardedjops, safe.ttt, ttt_ptypes, states, ttt_rls2 

Theory 

s n : VAR state 

safe_gwd.step: Lemma safe(akb,s n ) D safe(akb,g_win_dia(akb, s„)) 

Proof 

no.transition: Lemma 

(safe(akb, a n ) Ag_win_dia(3ikb ) s n ) = s n ) D safe(akb, g.win.dia(akb, s n )) 


ntp: Prove no.transition 
safe.trans: Lemma 

(safe(akb, s n ) A g_win_dia(akb, s n ) = next(s„)) 

D safe(akb, g_winjdia(akb, s n )) 

stll: Lemma 

g_win.dia(akb, a n ) = next(s n ) 

D (winjdiaJhs(akb, s n ) A winjdia(akb, s n ) = next(a n )) 

stllp: Prove stll from g_win_diajdef {s„ <— s n @cs} 

stl2: Lemma win_dia(akb, s n ) = next(s rt ) D won.game(akb,winjdia(akb, s n )) 

stl2p: Prove stl2 from 

winjdiajdef {«i «— s n @cs, p <— pi@pls}, 
win_dia_def {si *— s„@cs, p <— p 2 @pls}, 
won_game_def {kbx akb, 
s n *— next(s„@cs), 

Pi «- Pi@pl3, 
p 2 ♦- P 2 @pls, 

PS «“ P4@pls}, 
distinct jconstants 

stpp: Prove safe.trans from 
stll {s n «— s n @CS}, 
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stl2 {«„ «— s n @CS}, 

safejdef {kbx «— akb, a n <— win_dia(akb, s n @cs)} 

sgsll: Lemma win_dia(akb, s n ) = s n V winjdia(akb, s n ) = next(s n ) 

sgsllp: Prove sgsll from win_dia_def {«i +— « n @cs} 

safe_gwdjstep_proof: Prove safe_gwd_step from 
no.transition {s n <— s n @cs}, 
safe.tr ans {s n <— s n @cs}, 
sgsll {»„ <- a„@cs}, 
g.wimdiajdef {s n s n @cs} 

End gwd_proof 
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(defun remember (new) 

"Places a fact on the *facts* list if it is not already there.” 
(cond ((member new ’•‘facts* :test #’ equal) nil) 

(t (setf *facts* (cons new *facts*)) 
new) ) ) 

(defun recall (fact) 

"Tests to see if a fact is already asserted on the *facts* list." 
(cond ((member fact *facts* :test #’ equal) fact) 

(t nil))) 

(defun failed (fact) 

"Tests to see if a previous attempt to assert the fact failed." 
(cond ((and (not (recall fact)) 

(member fact *asked* :test #’equal)) 
t) 

(t nil))) 

(defun testif+ (rule) 

"Attempts to verify if each condition in the ifs clause 
can be satisfied.” 

(prog (ifs) 

(setf ifs (cdaddr rule)) 

(return 
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(loop 

(cond ((null ifs) (return t)) 

((if (eql (caar if 8) ’not) 

(not (verify (cadar ifs))) 

(verify (car ifs))) 

(setf ifs (cdr ifs))) 

(t (return nil))))))) 

(defun testif (rule) 

"Tests to see if each conditions in the ifs clause 
has already been asserted.” 

(prog (ifs) 

(setf ifs (cdaddr rule)) 

(return 

(loop 

(cond ((null if s) (return t)) 

((if (eql (caar ifs) ’not) 

(failed (cadar ifs)) 

(recall (car ifs))) 

(setf ifs (cdr ifs))) 

(t (return nil))))))) 

(defun inthen (fact) 

"Returns a list of rules for which fact occurs on the RHS." 
(mapcan #’ (lambda (r) 

(cond ((thenp fact r) 

(list r)))) 

*rules*)) 

(defun thenp (fact rule) 

"Test to see if a fact is in the RHS of a rule." 

(member fact (cadddr rule) :test #’ equal)) 

(defun usethen (rule) 

"Asserts the RHS of a rule whose LHS has been justified." 
(let ((thens (cdr (cadddr rule))) (success nil)) 

(loop 

(cond ((null thens) (return success)) 

((remember (car thens)) 



(format t ""%Rule <"A>"%"tdeduces: "A" 

(cadr rule) (car thens)) 

(setl success t) 

(set! thens (cdr thens))))))) 

(defun tryrule (rule) 

"Attempts to use a rule . " 

(and (testif rule) (usethen rule))) 

(defun tryrule+ (rule) 

"Attempts to use a rule. Uses a recursive call to verify, 
(and (testif + rule) (usethen rule))) 

(defmacro get-question (fact) 

'(cadr (assoc fact ^questions* :test #’ equal))) 

(defun verify (fact) 

"Main procedure which attempts to establish a fact." 

(prog (rl r2) 

(cond ((recall fact) (return t))) 

(setf rl (inthen fact)) 

(setf r2 rl) 

(if (null rl) 

(cond ((member fact *asked* :test #’ equal) (return nil)) 
((yes-or-no-p ""%"A?" (get-question fact)) 

(remember fact) 

(return t)) 

(t (setf *asked* (cons fact tasked*)) 

(return nil)))) 
loopl 

(cond ((null rl) 

(go loop2) ) 

((tryrule (car rl)) 

(return t))) 

(setf rl (cdr rl)) 

(go loopl) 
loop2 

(cond ((null r2) (go exit)) 

((tryrule+ (car r2)) 
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(return t))) 

(set! r2 (cdr r2)) 

(go loop2) 
exit 

(return nil))) 

(defun diagnose () 

"Top level function which conducts diagnosis." 

(let ((pos *hyps*) (*asked* nil) (*f acts* nil)) 

(declare (special *asked* *facts*)) 

(loop 

(cond ((null pos) 

(format t "~%No diagnosis available.") 

(return nil)) 

((verify (car pos)) 

(format t "”%Diagnosis: "A." (car pos)) 

(return (car pos)))) 

(setf pos (cdr pos))))) 

(defvar *rules* nil 

"The list of current rules in the system.") 

(def macro define -rule (fcrest rulebody) 

"A macro to make typing in rules more intuitive — Avoids hav 
ing to type in one long list of rules . " 

‘ (setf *rules* 

(append *rules* 

(list (cons ’rule ’ .rulebody))))) 

(defvar ^questions* nil 

"The list of queries to establish ground facts.") 

(defmacro phrase-question (fact question) 

"A macro to make typing in questions to be used to substanti 
ate facts more intuitive.” 

‘ (setf *questions* 

(cons (list ’ .fact .question) 

♦questions*))) 



(defun initialize () 

(set! *rules* nil) 

(set! ^questions* nil)) 

(defmacro hypothesis (ftrest hlist) 

"A macro to make entering hypothesis more intuitive. 
‘ (seti *hyps* hlist)) 
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Appendix F 

Heuristics for Electrical 
System Diagnosis 


small rule base lor diagnosis of problems in an automobile 
electrical system. 


(initialize) 

(deline-rule rl 

(il (engine wont start) 

(starter doesnt turn) 

(bad battery)) 

(then (replace or repair battery))) 

(phrase-question (engine wont start) 
"Does the engine FAIL to start") 
(phrase-question (starter doesnt turn) 
"Does the starter motor FAIL to turn") 

(deline -rule r2 

(il (engine wont start) 

(starter doesnt turn) 

(not (bad battery)) 

(jump solenoid starter turns) 
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(bad ignition)) 

(then (replace or repair ignition))) 

(phrase-question (jump solenoid starter turns) 

"Does the starter turn normally when a jumper is connected 
across the battery and starter posts of the solenoid") 

(define -rule r3 

(if (engine wont start) 

(starter doesnt turn) 

(not (bad battery)) 

(not (bad ignition)) 

(jump solenoid no response)) 

(then (replace or repair solenoid))) 

(phrase -question (jump solenoid no response) 

"Does the starter FAIL to turn at all when a jumper is 
connected across the battery and starter posts of the solenoid") 

(define -rule r4 

(if (engine wont start) 

(starter doesnt turn) 

(not (bad battery)) 

(jump solenoid starter turns)) 

(then (replace or repair starter))) 

(define -rule r5 

(if (engine wont start) 

(starter doesnt turn) 

(jump solenoid starter buzzes)) 

(then (replace or repair starter))) 

(phrase-question (jump solenoid starter buzzes) 

"Does the starter buzz or turn very slowly when a jumper 
is connected across the battery and starter posts of the 
solenoid") 


(define-rule r6 
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(if (battery case cracked)) 

(then (bad battery))) 

(phrase-question (battery case cracked) 

"Is the battery case cracked") 

(deline-rule r7 

(il (battery connections corroded)) 

(then (bad battery))) 

(phrase-question (battery connections corroded) 

"Are the battery clamps and posts corroded") 

(deline-rule r8 

(il (battery cells low)) 

(then (bad battery))) 

(phrase-question (battery cells low) 

"Is the state ol charge in any ol the cells lower 
than normal") 

(deline -rule rO 

(il (no power at ignition)) 

(then (bad ignition))) 

(phrase-question (no power at ignition) 

"Does the needle of a voltmeter connected to the start 
post ol the solenoid lail to move when the key is jiggled") 

(deline-rule rlO 

(il (llickers at ignition)) 

(then (bad ignition))) 

(phrase-question (llickers at ignition) 

"Does the needle ol a voltmeter connected to the start post 
ol the solenoid llicker when the key is jiggled") 

(hypothesis 

(replace or repair battery) 
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(replace or repair ignition) 
(replace or repair solenoid) 
(replace or repair starter)) 



Appendix G 

Derived Expert System 
Rules for Diagnosis 


simple expert system rule set lor diagnosis ol problems 
in a car’s electrical system. 


(initialize) 

(deline-rule dl 

(il (power at cells)) 

(then (power at battery))) 

(deline-rule dla 

(il (cell levels normal)) 

(then (power at cells))) 

(phrase-question (cell levels normal) 

"Are all ol the voltage levels ol the individual battery 
cells normal tt ) 

(deline-rule d2 

(il (power at battery) 

(good battery)) 

(then (power at ignition))) 
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(deline-rule d2a 

(il (not (bad battery case)) 

(not (bad battery connections))) 

(then (good battery))) 

(phrase-question (bad battery case) 

"Are there any cracks or damage to the battery case") 

(phrase-question (bad battery connections) 

"Are the battery posts and cable clamps corroded") 

(deline-rule d3 

(il (power at ignition) 

(good ignition)) 

(then (power at points) 

(power at solenoid))) 

(deline-rule d3a 

(il (not (open ignition switch)) 

(not (sporadic ignition))) 

(then (good ignition))) 

(phrase-question (open ignition switch) 

"Does a voltmeter connected to the starter post ol the 
solenoid lail to move when the key is turned to the start position") 
(phrase-question (sporadic ignition) 

"Does the needle ol a voltmeter llicker when the key is 
jiggled") 

(deline-rule d4 

(il (power at points) 

(good points)) 

(then (power at coil))) 

(deline -rule d4a 

(il (not (point gap oil)) 

(not (damaged points))) 

(then (good points))) 
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(phrase-question (point gap oil) 

"Does a dwell meter indicate the point gap needs adjusting") 
(phrase-question (damaged points) 

"Do the points show pitting, excessive wear, or other signs 
ol damage”) 

(deline-rule d5 

(il (power at coil) 

(good coil)) 

(then (power at distributor))) 

(deline -rule d5a 

(il (good primary coil resistance) 

(good secondary coil resistance)) 

(then (good coil))) 

(phrase-question (good primary coil resistance) 

"Does an ohmmeter indicate Irom 1 to 4 ohms ol resistance 
in the primary coil") 

(phrase -question (good secondary coil resistance) 

"Does an ohmmeter indicate Irom 4,000 to 10,000 ohms ol 
resistance in the secondary coil") 

(deline -rule d0 

(il (power at distributor) 

(good distributor)) 

(then (power at plugs))) 

(deline-rule d6a 

(il (not (bad distributor cap))) 

(then (good distributor))) 

(phrase-question (bad distributor cap) 

"Does the distributor cap and rotor show signs ol burning 
or corrosion") 

(deline-rule d7 

(il (power at solenoid) 

(good solenoid)) 
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(then (power at starter))) 

(define -rale d7a 

(if (jumped solenoid turns normal)) 

(then (good solenoid))) 

(phrase-question (jumped solenoid turns normal) 

"Does the starter turn normally when a jumper is connected 
between the battery and starter posts of the solenoid") 

(define-rule d7b 

(if (jumped solenoid starter buzz)) 

(then (good solenoid))) 

(phrase -question (jumped solenoid starter buzz) 

"Does the starter buzz or turn the engine slowly when a jumper 
is connected between the battery and starter posts of the 
solenoid" ) 

(define-rule d7c 

(if (not (jumped solenoid no response))) 

(then (good solenoid))) 

(phrase-question (jumped solenoid no response) 

"Does the starter show no response when a jumper is connected 
between the battery and starter posts of the solenoid") 

(define-rule d8 

(if (car fails to start) 

(starter does not turn)) 

(then (no start))) 

(phrase-question (car fails to start) 

"Does your car fail to start") 

(phrase-question (starter does not turn) 

"Does the starter fail to turn when the key is engaged") 

(define-rule yl 

(if (car fails to start) 



(not (starter does not turn)) 
(power at battery) 

(power at ignition) 

(power at points) 

(power at coil) 

(power at distributor) 

(power at plugs) 

(not (good plugs))) 

(then (replace plugs))) 

(phrase -question (good plugs) 

"Are the spark plugs undamaged") 

(define -rule y2 

(if (car fails to start) 

(not (starter does not turn)) 
(power at battery) 

(power at ignition) 

(power at points) 

(power at coil) 

(power at distributor) 

(not (good distributor))) 
(then (replace distributor))) 

(define-rule y3 

(if (car fails to start) 

(not (starter does not turn)) 
(power at battery) 

(power at ignition) 

(power at points) 

(power at coil) 

(not (good coil))) 

(then (replace coil))) 

(define-rule y4 

(if (car fails to start) 

(not (starter does not turn)) 
(power at battery) 

(power at ignition) 
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(power at points) 

(not (good points))) 

(then (replace points))) 

(define-mle y5 

(if (car fails to start) 

(starter does not turn) 

(power at battery) 

(power at ignition) 

(not (good ignition))) 

(then (replace ignition))) 

(define-rule y6 

(if (car fails to start) 

(starter does not turn) 

(power at battery) 

(not (good battery))) 

(then (replace battery))) 

(define-rule y7 

(if (car fails to start) 

(starter does not turn) 

(not (power at battery))) 

(then (replace battery))) 

(define-rule y8 

(if (car fails to start) 

(starter does not turn) 

(power at battery) 

(power at ignition) 

(power at solenoid) 

(power at starter) 

(not (good starter))) 

(then (replace starter))) 

(phrase-question (good starter) 

"Does the starter meet specifications when removed 
and bench tested") 



(deline-rule y9 

(il (car fails to start) 
(starter does not turn) 
(power at battery) 
(power at ignition) 
(power at solenoid) 

(not (good solenoid))) 
(then (replace solenoid))) 

(hypothesis 
(replace starter) 

(replace solenoid) 

(replace plugs) 

(replace distributor) 
(replace coil) 

(replace points) 

(replace ignition) 

(replace battery)) 
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Appendix H 

Electrical System Formal 
Specification 
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components: Module 

Exporting battery, cells, ignition, points, coil, distributor, 
plugs, solenoid, starter, component 

Theory 

component: TYPE 

battery, cells, ignition, points, coil, distributor, plugs, solenoid, 
starter: component 
c: VAR component 

enum: Axiom 

( V c: c = battery 
V c = cells 
Vc = ignition 
V c = points 
V c = coil 

Vc = distributor Vc = plugs Vc = solenoid Vc = starter) 


unique: Axiom 
battery ^ cells 

A battery ^ ignition 
A battery ^ points 
A battery ^ coil 
A battery ^ distributor 
A battery # plugs 
A battery ^ solenoid 
A battery # starter 
A cells ^ ignition 
A cells # points 
A cells ^ coil 
A cells 7^ distributor 
A cells 7^ plugs 
A cells 7^ solenoid 
A cells starter 
A ignition ^ points 
A ignition # coil 
A ignition ^ distributor 
A ignition # plugs 



A ignition ^ solenoid 
A ignition ^ starter 
A points 7^ coil 
A points ^ distributor 
A points # plugs 
A points 7^ solenoid 
A points ?£ starter 
A coil distributor 
A coil 7^ plugs 
A coil ^ solenoid 
A coil starter 
A distributor ^ plugs 
A distributor # solenoid 
A distributor ^ starter 
A plugs ^ solenoid 
A plugs 7^ starter 
A solenoid ^ starter 


End components 
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engine-model: Module 
Using components 
Exporting power _at, good 
Theory 

cl,c2: VAR component 

power_at: function [component — ► bool] 

good: function[component — ► bool] 

electrical-system_model: Axiom 

(power -at(battery) -O- good(cells)) 

A (power _at(ignition) O- power-at(battery) A good (battery)) 

A (power_at (points) <$■ power_at (ignition) A good(ignition)) 

A (power-at(coil) power_at (points) A good(points)) 

A (power _at (distributor) power_at(coil) A good(coil)) 

A(power_at (plugs) -O- power_at(distributor)Agood(distributor)) 

A(power-at(solenoid) <$■ power_at(ignition)Agood(ignition)) 

A(power_at(starter) -O- power_at(solenoid)Agood(solenoid)) 


single-failure: Axiom ->good(cl) D ( V c2: c2 ^ cl D good(c2)) 

Proof 

testl: Lemma -igood(coil) 3 ->power_at(distributor) A -power _at(plugs) 

tip: Prove testl from electrical _system_model 

test2: Lemma -ipower_at (ignition) D -ipower_at (starter) 

t2p: Prove test2 from electrical _system_model 

test3: Lemma power _at (coil) D power_at(battery) A good(ignition) 

t3p: Prove test3 from electrical _system_model 

End engine-model 



test: Module 

Using components, engine_model 
Theory 

batt: Lemma power_at(battery) 
igni: Lemma power_at(ignition) 
poin: Lemma power_at (points) 
coil: Le mm a power _at (coil) 
dist: Lemma power _at(distributor) 
plug: Lemma power_at(plugs) 
sole: Lemma power.at(solenoid) 
star: Lemma power_at (starter) 
End test 
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predict: Module 

Using components, engine-model, test 

Theory 

comp: component = solenoid 
cause: Axiom -igood(comp) 

Proof 

power-at_batt: Prove test.batt from 
cause, 

electrical jsystemjnodel, 
unique, 

single-failure {cl *— comp, c2 <— cells}, 
single-failure {cl «— comp, c2 <— battery}, 
single-failure {cl <— comp, c2 «— ignition}, 
single-failure {cl «— comp, c2 <— points}, 
single-failure {cl <— comp, c2 *— components.coil}, 
single-failure {cl <— comp, c2 ♦— distributor}, 
single-failure {cl <— comp, c2 <— plugs}, 
single-failure {cl «— comp, c2 +— solenoid}, 
single-failure {cl «— comp, c2 «— starter} 

power_at _igni: Prove test.igni from 
cause, 

electrical -system_model , 
unique, 

single-failure {cl <— comp, c2 «— cells}, 
single-failure {cl <— comp, c2 *— battery}, 
single-failure {cl <— comp, c2 *— ignition}, 
single-failure {cl *— comp, c2 <— points}, 
single-failure {cl «— comp, c2 *— components.coil}, 
single-failure {cl ♦— comp, c2 *— distributor}, 
single-failure {cl «— comp, c2 +- plugs}, 
single-failure {cl *— comp, c2 *— solenoid}, 
single-failure {cl ♦— comp, c2 ♦— starter} 

power-at_poin: Prove test.poin from 
cause, 

electrical jsystem_model, 



unique, 

single-failure {cl <— comp, c2 <— cells}, 
single-failure {cl <— comp, c2 <— battery), 
single-failure {cl «— comp, c2 <— ignition}, 
single-failure {cl comp, c2 «— points}, 
single-failure {cl <— comp, c2 <— components.coil}, 
single-failure {cl <— comp, c2 <— distributor}, 
single-failure {cl <— comp, c2 <— plugs}, 
single-failure {cl «— comp, c2 <— solenoid}, 
single Jailure {cl <— comp, c2 +— starter} 

power .at jcoil: Prove test .coil from 
cause, 

electrical _system_model, 
unique, 

single-failure {cl *— comp, c2 *— cells}, 
single Jailure {cl «— comp, c2 *- battery}, 
singleJailure {cl <— comp, c2 ♦- ignition}, 
singleJailure {cl comp, c2 <— points}, 
single-failure {cl *— comp, c2 <— components.coil}, 
single-failure {cl <— comp, c2 <— distributor}, 
single-failure {cl <— comp, c2 <— plugs}, 
single-failure {cl <— comp, c2 <— solenoid}, 
single-failure {cl comp, c2 ♦— starter} 

power_atjdist: Prove test.dist from 
cause, 

electrical -system .model , 
unique, 

single-failure {cl <— comp, c2 <— cells}, 
single-failure {cl <— comp, c2 «— battery}, 
single Jailure {cl <— comp, c2 ignition}, 
single-fsulure {cl «— comp, c2 points}, 
single Jailure {cl *— comp, c2 +— components.coil}, 
single-failure {cl ♦— comp, c2 <— distributor}, 
single-failure {cl <— comp, c2 <— plugs}, 
single Jailure {cl +— comp, c2 ■*— solenoid}, 
single Jailure {cl <— comp, c2 *— starter} 

power_at-plug: Prove test.plug from 
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cause, 

electrical .system_model, 
unique, 

single-failure {cl <— comp, c2 <— cells}, 
singleJailure {cl <— comp, c2 <— battery}, 
singleJailure {cl <— comp, c2 <— ignition}, 
single Jailure {cl <— comp, c2 *— points}, 
singleJailure {cl <— comp, c2 «— components.coil}, 
singleJailure {cl +- comp, c2 <— distributor}, 
singleJailure {cl <— comp, c2 *— plugs}, 
singleJailure {cl «— comp, c2 «— solenoid}, 
singleJailure {cl <— comp, c2 <— starter} 

power.at.sole: Prove test.sole from 
cause, 

electrical .system-model, 
unique, 

singleJailure {cl <— comp, c2 <— cells}, 
singleJailure {cl <— comp, c2 <— battery}, 
singleJailure {cl +- comp, c2 «— ignition}, 
singleJailure {cl comp, c2 points}, 

singleJailure {cl <— comp, c2 components.coil}, 

singleJailure {cl comp, c2 ♦- distributor}, 
singleJailure {cl ♦- comp, c2 *— plugs}, 
singleJailure {cl •*— comp, c2 *— solenoid}, 
singleJailure {cl «— comp, c2 starter} 

power_at_star: Prove test.star from 
cause, 

electrical-system-model , 
unique, 

singleJailure {cl <— comp, c2 *— cells}, 
singleJailure {cl «— comp, c2 ■<— battery}, 
singleJailure {cl *— comp, c2 «— ignition}, 
singleJailure {cl *- comp, c2 *- points}, 
singleJailure {cl ♦- comp, c2 *— components.coil}, 
singleJailure {cl <— comp, c2 *- distributor}, 
single Jjulure {cl «— comp, c2 *— plugs}, 
singleJailure {cl *— comp, c2 *- solenoid}, 
singleJailure {cl <— comp, c2 ♦— starter} 



End predict 
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